Posted on

Microsoft launches Open Service Mesh

Microsoft today announced the launch of a new open-source service mesh based on the Envoy proxy. The Open Service Mesh is meant to be a reference implementation of the Service Mesh Interface (SMI) spec, a standard interface for service meshes on Kubernetes that has the backing of most of the players in this ecosystem.

The company plans to donate Open Service Mesh to the Cloud Native Computing Foundation (CNCF) to ensure that it is community-led and has open governance.

“SMI is really resonating with folks and so we really thought that there was room in the ecosystem for a reference implementation of SMI where the mesh technology was first and foremost implementing those SMI APIs and making it the best possible SMI experience for customers,” Microsoft partner program manager (and CNCF board member) Gabe Monroy told me.

Image Credits: Microsoft

He also added that, because SMI provides the lowest common denominator API design, Open Service Mesh gives users the ability to “bail out” to raw Envoy if they need some more advanced features. This “no cliffs” design, Monroy noted, is core to the philosophy behind Open Service Mesh.

As for its feature set, SMI handles all of the standard service mesh features you’d expect, including securing communications between services using mTLS, managing access control policies, service monitoring and more.

Image Credits: Microsoft

There are plenty of other service mesh technologies in the market today, though. So why would Microsoft launch this?

“What our customers have been telling us is that solutions that are out there today, Istio being a good example, are extremely complex,” he said. “It’s not just me saying this. We see the data in the AKS support queue of customers who are trying to use this stuff — and they’re struggling right here. This is just hard technology to use, hard technology to build at scale. And so the solutions that were out there all had something that wasn’t quite right and we really felt like something lighter weight and something with more of an SMI focus was what was going to hit the sweet spot for the customers that are dabbling in this technology today.”

Monroy also noted that Open Service Mesh can sit alongside other solutions like Linkerd, for example.

A lot of pundits expected Google to also donate its Istio service mesh to the CNCF. That move didn’t materialize. “It’s funny. A lot of people are very focused on the governance aspect of this,” he said. “I think when people over-focus on that, you lose sight of how are customers doing with this technology. And the truth is that customers are not having a great time with Istio in the wild today. I think even folks who are deep in that community will acknowledge that and that’s really the reason why we’re not interested in contributing to that ecosystem at the moment.”

Read More

Posted on

LA-based Replicated adds former GitLab head of product as its chief product officer

Replicated, the Los Angeles-based company pitching monitoring and management services for Kubernetes-based applications, has managed to bring on the former head of product of the $2.75 billion-valued programming giant GitLab as its new chief product officer. 

Mark Pundsack is joining the company as it moves to scale its business. At GitLab, Pundsack saw the company grow from 70 employees to 1,300 as it scaled its business through its on-premise offerings.

Replicated is hoping to bring the same kind of on-premise services to a broad array of enterprise clients, according to company chief executive Grant Miller.

First introduced to Replicated while working with CircleCI, it was the company’s newfound traction since the launch of its Kubernetes deployment management toolkit that caused him to take a second look.

“The momentum that Replicated has created with their latest offering is tremendous; really changing the trajectory of the company,” said Pundsack in a statement. “When I was able to get close to the product, team, and customers, I knew this was something that I wanted to be a part of. This company is in such a unique position to create value throughout the entire enterprise software ecosystem; this sort of reach is incredibly rare. The potential reminds me a lot of the early days of GitLab.”

It’s a huge coup for Replicated, according to Miller.

“Mark created the core product strategy at GitLab; transforming GitLab from a source control company to a complete DevOps platform, with incredible support for Kubernetes,” said Miller. “There really isn’t a better background for a product leader at Replicated; Mark has witnessed GitLab’s evolution from a traditional on-prem installation towards a Kubernetes-based installation and management experience. This is the same transition that many of our customers are going through and Mark has already done it with one of the best. I have so much confidence that his involvement with our product will lead to more success for our customers.”

Pundsack is the second new executive hire from Replicated in six months, as the company looks to bring more muscle to its C-suite and expand its operations.

Read More

Posted on

Scaleway launches managed Kubernetes clusters

Cloud hosting company Scaleway has launched Kubernetes Kapsule, a new service that lets you manage Kubernetes clusters on Scaleway’s infrastructure. The service works with a wide-range of Scaleway instances and lets you create large clusters that scales depending on demand.
Kubernetes is an open-source platform to manage containers and …

Read More

Posted on

Spectro Cloud launches with $7.5M investment to help developers build Kubernetes clusters their way

By now we know that Kubernetes is a wildly popular container management platform, but if you want to use it, you pretty much have to choose between having someone manage it for you or building it yourself. Spectro Cloud emerged from stealth today with a $7.5 million investment to give you …

Read More

Posted on

Kubernetes Going Mainstream — Innovative Startups Rise

VMWare describes the shift to the cloud as the most significant shift in enterprise architecture of the decade and that 500 million new apps will be launched in the next five years. Thanks to containers, compute costs are falling by 50%, and resources can be provisioned 450X faster. With Kubernetes going mainstream — …

Read More

Posted on

Simplify Machine Learning Inference on Kubernetes with Amazon SageMaker Operators

Amazon SageMaker Operators for Kubernetes allows you to augment your existing Kubernetes cluster with SageMaker hosted endpoints.

Machine learning inferencing requires investment to create a reliable and efficient service. For an XGBoost model, developers have to create an application, such as through Flask that will load the model and then run the endpoint, which requires developers to think about queue management, faultless deployment, and reloading of newly trained models. The serving container then has to be pushed to a Docker repository, where Kubernetes can be configured to pull from, and deploy on the cluster. These steps require your data scientist to work on tasks unrelated to improving model accuracy, or bringing in a dev ops engineer, which adds to development schedules and requires more time to iterate.

With the SageMaker Operators, developers only need to write a yaml file that specifies the S3 stored locations of the saved models, and live predictions become available through a secure endpoint. Reconfiguring the endpoint is as simple as updating the yaml file. On top of being easy to use, the service also has the following features:

  • Multi-model endpoint – Hosting dozens or more models can be challenging to configure and lead to many machines operating with low utilization. Multi-model endpoints sets up one instance with on the fly loading of model artifacts for serving
  • Elastic Inference – Run your smaller workloads on split GPUs that you can deploy at low cost
  • High Utilization & Dynamic Auto Scaling – Endpoints can run with 100% utilization and add replicas based on custom metrics you define, such as invocations per second. Alternatively, automatic scaling can be configured on predefined metrics for client performance
  • Availability Zone Transfer – If there is an outage, Amazon SageMaker will automatically move your endpoint to another Availability Zone within your VPC
  • A/B Testing – Set up multiple models, and direct traffic proportional to the amount that you set on a single endpoint
  • Security – Endpoints are created with HTTPS and can be configured to be run in a private VPC (no internet egress) and accessed through AWS PrivateLink
  • Compliance Ready – Amazon SageMaker has been certified compliant with HIPAA, PCI DSS, and SOC (1, 2, 3) rules and regulations

Packaged together, the features that are available in Kubernetes through SageMaker Operators shorten time to launch model serving, and reduce your development resources to setup and maintain production infrastructure. This can be a drop of 90% in total cost of ownership over EKS or EC2 alone.

This post demonstrates how to set up Amazon SageMaker Operators for Kubernetes to create and update endpoints for a pre-trained XGBoost model completely from kubectl. The solution contains the following steps:

  • Create an IAM Amazon SageMaker role, which gives Amazon SageMaker permissions needed to serve your model
  • Prepare a YAML file that deploys your model to Amazon SageMaker
  • Deploy your model to Amazon SageMaker
  • Query the endpoint to obtain predictions
  • Perform an eventually consistent update to the deployed model

Prerequisites

This post assumes you have the following prerequisites:

  • A Kubernetes cluster
  • The Amazon SageMaker Operators installed on your cluster
  • An XGBoost model you can deploy

For information about installing the operator onto an Amazon EKS cluster, see Introducing Amazon SageMaker Operators for Kubernetes. You can bring your own XGBoost model, but this tutorial uses the existing model from the previously mentioned post.

Creating an Amazon SageMaker execution role

Amazon SageMaker needs an IAM role that it can assume to serve your model. If you do not have one already, create one with the following bash code:

export assume_role_policy_document='{
  "Version": "2012-10-17",
  "Statement": [{
    "Effect": "Allow",
    "Principal": {
      "Service": "sagemaker.amazonaws.com"
    },
    "Action": "sts:AssumeRole"
  }]
}'
aws iam create-role --role-name <execution role name> 
    --assume-role-policy-document 
    "$assume_role_policy_document"
aws iam attach-role-policy --role-name <execution role name> 
    --policy-arn 
    arn:aws:iam::aws:policy/AmazonSageMakerFullAccess

Replace <execution role name> with a suitable role name. This creates an IAM role that Amazon SageMaker can use to assume when serving your model.

Preparing your hosting deployment

The operators provide a Custom Resource Definition (CRD) named HostingDeployment. You use a HostingDeployment to configure your model deployment on Amazon SageMaker Hosting.

To prepare your hosting deployment, create a file called hosting.yaml with the following contents:

apiVersion: sagemaker.aws.amazon.com/v1
kind: HostingDeployment
metadata:
  name: hosting-deployment
spec:
  region: us-east-2
  productionVariants:
    - variantName: AllTraffic
      modelName: xgboost-model
      initialInstanceCount: 1
      instanceType: ml.r5.large
      initialVariantWeight: 1
  models:
    - name: xgboost-model
      executionRoleArn: SAGEMAKER_EXECUTION_ROLE_ARN
      containers:
        - containerHostname: xgboost
          modelDataUrl: s3://BUCKET_NAME/model.tar.gz
          image: 825641698319.dkr.ecr.us-east-2.amazonaws.com/xgboost:latest

Replace SAGEMAKER_EXECUTION_ROLE_ARN with the ARN of the execution role you created in the previous step. Replace BUCKET_NAME with the bucket that contains your model.

Make sure that the bucket Region, HostingDeployment Region, and image ECR Region are all equivalent.

Deploying your model to Amazon SageMaker

You can now start the deployment by running kubectl apply -f hosting.yaml. See the following code:

$ kubectl apply -f hosting.yaml
hostingdeployment.sagemaker.aws.amazon.com/hosting-deployment created

You can track deployment status with kubectl get hostingdeployments. See the following code:

$ kubectl get hostingdeployments
NAME                 STATUS     SAGEMAKER-ENDPOINT-NAME
hosting-deployment   Creating   hosting-deployment-38ecac47487611eaa81606fc3390e6ba

Your model endpoint may take up to fifteen minutes to be deployed.  You can use the below command to view the status.  The endpoint will be ready for queries once it achieves the InService status.

$ kubectl get hostingdeployments
NAME                 STATUS      SAGEMAKER-ENDPOINT-NAME
hosting-deployment   InService   hosting-deployment-38ecac47487611eaa81606fc3390e6ba

Querying the endpoint

After the endpoint is in service, you can test that it works with the following example code:

$ aws sagemaker-runtime invoke-endpoint 
  --region us-east-2 
  --endpoint-name SAGEMAKER-ENDPOINT-NAME 
  --body $(seq 784 | xargs echo | sed 's/ /,/g') 
  >(cat) 
  --content-type text/csv > /dev/null

This bash command connects to the HTTPS endpoint using AWS CLI. The model you created is based on the MNIST digit dataset, and your predictor reads what number is in the image. When you make this call, it sends an inference payload that contains 784 features in CSV format, which represent pixels in an image. You see the predicted number that the model believes is in the payload. See the following code:

$ aws sagemaker-runtime invoke-endpoint 
  --region us-east-2 
  --endpoint-name hosting-deployment-38ecac47487611eaa81606fc3390e6ba 
  --body $(seq 784 | xargs echo | sed 's/ /,/g') 
  >(cat) 
  --content-type text/csv > /dev/null
8.0

This confirms that your endpoint is up and running.

Eventually consistent updates

After you deploy a model, you can make changes to the Kubernetes YAML and the operator updates the endpoint. The updates propagate to Amazon SageMaker in an eventually consistent way. This enables you to configure your endpoints declaratively and lets the operator handle the details.

To demonstrate this, you can change the instance type of the model from ml.r5.large to ml.c5.2xlarge. Complete the following steps:

  1. Modify the instance type in hosting.yaml to be ml.c5.2xlarge. See the following code:
    apiVersion: sagemaker.aws.amazon.com/v1
    kind: HostingDeployment
    metadata:
      name: hosting-deployment
    spec:
      region: us-east-2
      productionVariants:
        - variantName: AllTraffic
          modelName: xgboost-model
          initialInstanceCount: 1
          instanceType: ml.c5.2xlarge
          initialVariantWeight: 1
      models:
        - name: xgboost-model
          executionRoleArn: SAGEMAKER_EXECUTION_ROLE_ARN
          containers:
            - containerHostname: xgboost
              modelDataUrl: s3://BUCKET_NAME/model.tar.gz
              image: 825641698319.dkr.ecr.us-east-2.amazonaws.com/xgboost:latest
    
  2. Apply the change to the Kubernetes cluster. See the following code:
    $ kubectl apply -f hosting.yaml
    hostingdeployment.sagemaker.aws.amazon.com/hosting-deployment configured

  3. Get the status of the hosting deployment. It will show as Updating and then change to InService when ready. See the following code:
    $ kubectl get hostingdeployments
    NAME                 STATUS     SAGEMAKER-ENDPOINT-NAME
    hosting-deployment   Updating   hosting-deployment-38ecac47487611eaa81606fc3390e6ba

The endpoint remains live and fully available throughout the update. For more information and additional examples, see the GitHub repo.

Cleaning up

To delete the endpoint, and not incur further usage charges, run kubectl delete -f hosting.yaml. See the following code:

$ kubectl delete -f hosting.yaml
hostingdeployment.sagemaker.aws.amazon.com "hosting-deployment" deleted

Conclusion

This post demonstrated how Amazon SageMaker Operators for Kubernetes supports real-time inference. It also supports training and hyperparameter tuning.

As always, please share your experience and feedback, or submit additional example YAML specs or operator improvements. You can share how you’re using Amazon SageMaker Operators for Kubernetes by posting on the AWS forum for Amazon SageMaker, creating issues in the GitHub repo, or sending it through your AWS Support contacts.


About the authors

Cade Daniel is a Software Development Engineer with AWS Deep Learning. He develops products that make training and serving DL/ML models more efficient and easy for customers. Outside of work, he enjoys practicing his Spanish and learning new hobbies.

Alex Chung is a Senior Product Manager with AWS in Deep Learning. His role is to make AWS Deep Learning products more accessible and cater to a wider audience. He’s passionate about social impact and technology, getting his regular gym workout, and cooking healthy meals.

Source: AWS Machine Learning Blog

Posted on

Epsagon scores $16M Series A to monitor modern development environments

Epsagon, an Israeli startup that wants to help monitor modern development environments like serverless and containers, announced a $16 million Series A today.

U.S. Venture Partners (USVP), a new investor led the round. Previous investors Lightspeed Venture Partners and StageOne Ventures also participated. Today’s investment brings the total raised to $20 million, according to the company.

CEO and co-founder Nitzan Shapira says that the company has been expanding its product offerings in the last year to cover not just its serverless roots, but also giving deeper insights into a number of forms of modern development.

“So we spoke around May when we launched our platform for microservices in the cloud products, and that includes containers, serverless and really any kind of workload to build microservices apps. Since then we have had a few several significant announcements,” Shapira told TechCrunch.

For starters, the company announced support or tracing and metrics for Kubernetes workloads including native Kubernetes along with managed Kubernetes services like AWS EKS and Google GKE. “A few months ago, we announced our Kubernetes integration. So, if you’re running any Kubernetes workload, you can integrate with Epsagon in one click, and from there you get all the metrics out of the box, then you can set up a tracing in a matter of minutes. So that opens up a very big number of use cases for us,” he said.

The company also announced support for AWS AppSync, a no-code programming tool on the Amazon cloud platform. “We are the only provider today to introduce tracing for AppSync and that’s [an area] where people really struggle with the monitoring and troubleshooting of it,” he said.

The company hopes to use the money from today’s investment to expand the product offering further with support for Microsoft Azure and Google Cloud Platform in the coming year. He also wants to expand the automation of some tasks that have to be manually configured today.

“Our intention is to make the product as automated as possible, so the user will get an amazing experience in a matter of minutes including advanced monitoring, identifying different problems and troubleshooting,” he said

Shapira says the company has around 25 employees today, and plans to double headcount in the next year.

Source: TechCrunch