Kafka connect schema registry

Kafka connect schema registry DEFAULT

Avro Serialization

A Debezium connector works in the Kafka Connect framework to capture each row-level change in a database by generating a change event record. For each change event record, the Debezium connector completes the following actions:

  1. Applies configured transformations.

  2. Serializes the record key and value into a binary form by using the configured Kafka Connect converters.

  3. Writes the record to the correct Kafka topic.

You can specify converters for each individual Debezium connector instance. Kafka Connect provides a JSON converter that serializes the record keys and values into JSON documents. The default behavior is that the JSON converter includes the record’s message schema, which makes each record very verbose. The Debezium Tutorial shows what the records look like when both payload and schemas are included. If you want records to be serialized with JSON, consider setting the following connector configuration properties to :

Setting these properties to excludes the verbose schema information from each record.

Alternatively, you can serialize the record keys and values by using Apache Avro. The Avro binary format is compact and efficient. Avro schemas make it possible to ensure that each record has the correct structure. Avro’s schema evolution mechanism enables schemas to evolve. This is essential for Debezium connectors, which dynamically generate each record’s schema to match the structure of the database table that was changed. Over time, change event records written to the same Kafka topic might have different versions of the same schema. Avro serialization makes it easier for the consumers of change event records to adapt to a changing record schema.

To use Apache Avro serialization, you must deploy a schema registry that manages Avro message schemas and their versions. Available options include the Apicurio API and Schema Registry as well as the Confluent Schema Registry. Both are described here.

Sours: https://debezium.io/documentation/reference/configuration/avro.html

Avro Serialization

Debezium connectors are used with the Kafka Connect framework to capture changes in databases and generate change events. The Kafka Connect workers then apply to each of the messages generated by the connector the transformations configured for the connector, serialize each message key and value into a binary form using the worker’s converters, and finally write each messages into the correct Kafka topic.

The converters are specified in the Kafka Connect worker configuration, and the same converters are used for all connectors deployed to that worker’s cluster. Kafka Connect comes with a JSON converter that serializes the message keys and values into JSON documents. The JSON converter can be configured to include or exclude the message schema using the ( and ) properties. Our tutorial shows what the messages look like when both payload and schemas are included, but the schemas make the messages very verbose. If you want your messages serialized with JSON, consider setting these properties to to exclude the verbose schema information.

Another option is to serialize the message keys and values using Apache Avro. The Avro binary format is extremely compact and efficient, and Avro schemas make it possible to ensure that the messages have the correct structure. Avro’s schema evolution mechanism makes it possible to evolve the schemas over time, which is essential for Debezium connectors that dynamically generate the message schemas to match the structure of the database tables. Over time, the change events captured by Debezium connectors and written by Kafka Connect into a topic may have different versions of the same schema, and Avro serialization makes it far easier for consumers to adapt to the changing schema.

  • An Avro Converter that can be used in Kafka Connect workers to map the Kafka Connect schemas into Avro schemas and to then use those Avro schemas to serialize the message keys and values into the very compact Avro binary form.

  • A Schema Registry that tracks all of the Avro schemas used in Kafka topics, and where the Avro Converter sends the generated Avro schemas. Since the Avro schemas are stored in this registry, each message need only include a tiny schema identifier. This makes each message even smaller, and for an I/O bound system like Kafka this means more total throughput of the producers and consumers.

  • Avro Serdes (serializers and deserializers) for Kafka producers and consumers. Any Kafka consumer applications you write to consume change events can use the Avro serdes to deserialize the changes events.

These Confluent components are open source, and you can install them into any Kafka distribution and use them with Kafka Connect. However, Confluent also provides a Confluent Open Source Platform that includes the standard Kafka distribution as well as these and other Confluent open source components, including several source and sink connectors. Some Docker images for Kafka Connect also contain the Avro converter. This includes recent Debezium Docker images that include the Debezium connectors, Kafka Connect, and the Avro converter.

Sours: https://debezium.io/documentation/reference/0.10/configuration/avro.html
  1. Kodi exodus indigo error
  2. Skin act wax warmer
  3. Monkey with ak video

Using Kafka Connect with Schema Registry¶

Looking for Confluent Cloud Schema Management docs? These pages cover some aspects of Schema Registry that are generally applicable, such as general concepts, schema formats, hybrid use cases, and tutorials, but the main focus here is Confluent Platform. For Confluent Cloud documentation, check out Manage Schemas on Confluent Cloud.

Kafka Connect and Schema Registry integrate to capture schema information from connectors. Kafka Connect converters provide a mechanism for converting data from the internal data types used by Kafka Connect to data types represented as Avro, Protobuf, or JSON Schema. The , , and automatically register schemas generated by source connectors. Sink Connectors receive schema information in addition to the data for each message. This allows sink connectors to know the structure of the data to provide additional capabilities like maintaining a database table structure or creating a search index. Each of the converters change schema data into the internal data types used by Kafka Connect.

For additional information about converters and how they work, see Configuring Key and Value Converters.

Example Converter Properties¶

To use Kafka Connect with Schema Registry, you must specify the or properties in the connector or in the Connect worker configuration. The converters need an additional configuration for the Schema Registry URL, which is specified by providing the URL converter prefix as shown in the following property examples.

Avro¶

Example Avro converter properties are shown below:

key.converter=io.confluent.connect.avro.AvroConverterkey.converter.schema.registry.url=http://localhost:8081value.converter=io.confluent.connect.avro.AvroConvertervalue.converter.schema.registry.url=http://localhost:8081

The following additional configuration properties can be used with the Avro converter (). These Avro-specific properties are added to the worker or connector configuration where the Avro converter properties are located. Note that when added to the worker or connector configuration, these properties require the and prefix. For example:

key.converter=io.confluent.connect.avro.AvroConverterkey.converter.schema.registry.url=http://localhost:8081key.converter.enhanced.avro.schema.support=truevalue.converter=io.confluent.connect.avro.AvroConvertervalue.converter.schema.registry.url=http://localhost:8081value.converter.enhanced.avro.schema.support=true

Note that when using Avro in a secure environment, you need to add properties. An example of these additional properties is shown below:

key.converter.schema.registry.ssl.truststore.location=<location>key.converter.schema.registry.ssl.truststore.password=<trustore-password>key.converter.schema.registry.ssl.keystore.location=<keystore-location>key.converter.schema.registry.ssl.keystore.password=<keystore-password>key.converter.schema.registry.ssl.key.password=<key-password>value.converter.schema.registry.ssl.truststore.location=<location>value.converter.schema.registry.ssl.truststore.password=<trustore-password>value.converter.schema.registry.ssl.keystore.location=<keystore-location>value.converter.schema.registry.ssl.keystore.password=<keystore-password>value.converter.schema.registry.ssl.key.password=<key-password>

The following lists definitions for the Avro-specific properties shown in the examples above. For a complete list of Connect Schema Registry configuration options, see Configuration Options.

The size of the schema cache used in the Avro converter.

  • Type: int
  • Default: 1000
  • Importance: low

Enable enhanced Avro schema support in the Avro Converter. When set to , this property preserves Avro schema package information and Enums when going from Avro schema to Connect schema. This information is added back in when going from Connect schema to Avro schema.

  • Type: boolean
  • Default: false
  • Importance: low

Allow the Connect converter to add its metadata to the output schema.

  • Type: boolean
  • Default: true
  • Importance: low

The property preserves the following Connect schema metadata when going from Connect schema to Avro schema. The following metadata is added back in when going from Avro schema to Connect schema.

  • doc
  • version
  • parameters
  • default value
  • name
  • type

Protobuf¶

Protobuf example converter properties are shown below:

key.converter=io.confluent.connect.protobuf.ProtobufConverterkey.converter.schema.registry.url=http://localhost:8081value.converter=io.confluent.connect.protobuf.ProtobufConvertervalue.converter.schema.registry.url=http://localhost:8081

JSON Schema¶

JSON Schema example converter properties are shown below:

key.converter=io.confluent.connect.json.JsonSchemaConverterkey.converter.schema.registry.url=http://localhost:8081value.converter=io.confluent.connect.json.JsonSchemaConvertervalue.converter.schema.registry.url=http://localhost:8081

Using Independent Key and Value Converters¶

The key and value converters can be used independently from each other. For example, you may want to use a for keys and a converter used with Schema Registry for values. An example of independent key and value properties is shown below:

key.converter=org.apache.kafka.connect.storage.StringConvertervalue.converter=io.confluent.connect.avro.AvroConvertervalue.converter.schema.registry.url=http://localhost:8081

Converter Property Location and Inheritance¶

Confluent Platform first looks for converter configuration properties in the connector configuration. If none are found there, properties in the Connect worker configuration are used. You have the following three options for how to set these properties. Each one affects how the properties are inherited among the worker and connectors.

  • Specify all converter properties (with Schema Registry URL prefixes) in each connector configuration.
  • Specify all converter properties only in the worker configuration. In this case, all connectors inherit the worker converter properties.
  • Specify all converter properties in the worker configuration and add converter overrides in the connector configuration.

Important

  • If converter values and associated Schema Registry URL are defined in both the worker and the connector, settings in the connector overwrite those in the worker.
  • If you specify a converter in a connector or worker (as an override or as the only setting), you must always include both the converter and the Schema Registry URL, otherwise the connector or worker will fail.
  • If you specify a converter in a connecter that is not defined in the worker, must supply all converter properties (key converter, value converter, and Schema Registry host:port) in the connector configuration.

Example Scenario¶

The following are the worker configuration properties used in this example scenario:

group.id=connect-cluster key.converter=io.confluent.connect.avro.AvroConverter key.converter.schema.registry.url=http://host-1:port value.converter=org.apache.kafka.connect.storage.StringConverter

Using the previous worker properties, start three connectors with the following configuration properties:

  • connector-1 configuration:

    name=connector-1 <no converter configuration properties used>
  • connector-2 configuration:

    name=connector-2 key.converter=io.confluent.connect.avro.AvroConverter key.converter.schema.registry.url=http://host-2:port
  • connector-3 configuration:

    name=connector-3 key.converter=io.confluent.connect.avro.AvroConverter

The results of the deployment are:

  • connector-1 uses the worker configuration properties, with the Avro converter () and the Schema Registry .
  • connector-2 uses the Avro converter () and the Schema Registry .
  • connector-3 fails because it attempts to use the connector configuration, but does not find the Schema Registry URL configuration property. The Schema Registry URL configuration property is required for Avro, Protobuf, and JSON Schema.
  • All connectors use the worker property .

Configuration Options¶

Comma-separated list of URLs for Schema Registry instances that can be used to register or look up schemas.

  • Type: list
  • Default: “”
  • Importance: high

Specify if the Serializer should attempt to register the Schema with Schema Registry.

  • Type: boolean
  • Default: true
  • Importance: medium

Only applies when is set to . If is set to and is set to , then instead of deriving a schema for the object passed to the client for serialization, Schema Registry will use the latest version of the schema in the subject for serialization.

  • Type: boolean
  • Default: false
  • Importance: medium

Only applies when is set to .

If is (the default), then when using during serialization, a check is performed to verify that the latest subject version is backward compatible with the schema of the object being serialized. If the check fails, then an error results. If the check succeeds, then serialization is performed.

If is , then the latest subject version is used for serialization, without any compatibility check. Serialization may fail in this case. Relaxing the compatibility requirement (by setting to ) may be useful, for example, when implementing Kafka Connect converters and schema references.

  • Type: boolean
  • Default: true
  • Importance: medium

Maximum number of schemas to create or cache locally.

  • Type: int
  • Default: 1000
  • Importance: low

Determines how to construct the subject name under which the key schema is registered with Schema Registry. For additional information, see Schema Registry Subject Name Strategy.

Any implementation of can be specified. By default, is used as subject. Specifying an implementation of is deprecated as of and if used may have some performance degradation.

  • Type: class
  • Default: class io.confluent.kafka.serializers.subject.TopicNameStrategy
  • Importance: medium

Determines how to construct the subject name under which the value schema is registered with Schema Registry. For additional information, see Schema Registry Subject Name Strategy.

Any implementation of can be specified. By default, is used as subject. Specifying an implementation of is deprecated as of and if used may have some performance degradation.

  • Type: class
  • Default: class io.confluent.kafka.serializers.subject.TopicNameStrategy
  • Importance: medium

Specify how to pick the credentials for Basic authentication header. The supported values are URL, USER_INFO and SASL_INHERIT

  • Type: string
  • Default: “URL”
  • Importance: medium

Specify the user info for Basic authentication in the form of {username}:{password}. schema.registry.basic.auth.user.info is a deprecated alias for this configuration.

  • Type: password
  • Default: “”
  • Importance: medium

The following Schema Registry dedicated properties, configurable on the client, are available on Confluent Platform version 5.4.0 (and later). To learn more, see the information on configuring clients in Additional configurations for HTTPS.

The location of the trust store file. For example,

  • Type: string
  • Default: “”
  • Importance: medium

The password for the trust store file. If a password is not set, access to the truststore is still available but integrity checking is disabled.

  • Type: password
  • Default: “”
  • Importance: medium

The location of the key store file. This is optional for the client and can be used for two-way authentication for the client. For example, .

  • Type: string
  • Default: “”
  • Importance: medium

The store password for the key store file. This is optional for the client and only needed if is configured.

  • Type: password
  • Default: “”
  • Importance: medium

The password of the private key in the key store file. This is optional for the client.

  • Type: password
  • Default: “”
  • Importance: medium

© Copyright , Confluent, Inc. Privacy Policy | Terms & Conditions. Apache, Apache Kafka, Kafka and the Kafka logo are trademarks of the Apache Software Foundation. All other trademarks, servicemarks, and copyrights are the property of their respective owners.

Sours: https://docs.confluent.io/platform/current/schema-registry/connect.html

Using Kafka Connect with Schema Registry¶

Kafka Connect and Schema Registry integrate to capture schema information from connectors. Kafka Connect converters provide a mechanism for converting data from the internal data types used by Kafka Connect to data types represented as Avro, Protobuf, or JSON Schema. The , , and automatically register schemas generated by source connectors. Sink Connectors receive schema information in addition to the data for each message. This allows sink connectors to know the structure of the data to provide additional capabilities like maintaining a database table structure or creating a search index. Each of the converters change schema data into the internal data types used by Kafka Connect.

For additional information about converters and how they work, see Configuring Key and Value Converters.

Example Configuration¶

To use Kafka Connect with Schema Registry, you must specify the or properties in the connector or in the Connect worker configuration.

The converters need an additional configuration for the Schema Registry URL, which is specified by providing the required prefix.

Avro¶

Avro example properties are shown below:

key.converter=io.confluent.connect.avro.AvroConverterkey.converter.schema.registry.url=http://localhost:8081value.converter=io.confluent.connect.avro.AvroConvertervalue.converter.schema.registry.url=http://localhost:8081

Protobuf¶

Protobuf example properties are shown below:

key.converter=io.confluent.connect.protobuf.ProtobufConverterkey.converter.schema.registry.url=http://localhost:8081value.converter=io.confluent.connect.protobuf.ProtobufConvertervalue.converter.schema.registry.url=http://localhost:8081

JSON Schema¶

JSON Schema example properties are shown below:

key.converter=io.confluent.connect.json.JsonSchemaConverterkey.converter.schema.registry.url=http://localhost:8081value.converter=io.confluent.connect.json.JsonSchemaConvertervalue.converter.schema.registry.url=http://localhost:8081

Using Independent Key and Value Converters¶

The key and value converters can be used independently from each other. For example, you may want to use a for keys and a converter used with Schema Registry for values. An example of independent key and value properties is shown below:

key.converter=org.apache.kafka.connect.storage.StringConvertervalue.converter=io.confluent.connect.avro.AvroConvertervalue.converter.schema.registry.url=http://localhost:8081

Where to Specify Converter Configurations and How Properties are Inherited¶

Confluent Platform first looks for converter configuration properties in the connector configuration. If none are found there, properties in the Connect worker configuration are used. You have the following three options for how to set these properties. Each one affects how the properties are inherited among the worker and connectors.

  • Specify all converter properties (with Schema Registry URL prefixes) in each connector configuration.
  • Specify all converter properties only in the worker configuration. In this case, all connectors inherit the worker converter properties.
  • Specify all converter properties in the worker configuration and add converter overrides in the connector configuration.

Important

  • If converter values and associated Schema Registry URL are defined in both the worker and the connector, settings in the connector overwrite those in the worker.
  • If you specify a converter in a connector or worker (as an override or as the only setting), you must always include both the converter and the Schema Registry URL, otherwise the connector or worker will fail.
  • If you specify a converter in a connecter that is not defined in the worker, must supply all converter properties (key converter, value converter, and Schema Registry host:port) in the connector configuration.

Example Scenario¶

The following are the worker configuration properties used in this example scenario:

group.id=connect-clusterkey.converter=io.confluent.connect.avro.AvroConverterkey.converter.schema.registry.url=http://host-1:portvalue.converter=org.apache.kafka.connect.storage.StringConverter

Using the worker properties above, start three connectors with the following configuration properties:

  • connector-1 configuration:

    name=connector-1<noconverterconfigurationpropertiesused>
  • connector-2 configuration:

    name=connector-2key.converter=io.confluent.connect.avro.AvroConverterkey.converter.schema.registry.url=http://host-2:port
  • connector-3 configuration:

    name=connector-3key.converter=io.confluent.connect.avro.AvroConverter

The results of the deployment are:

  • connector-1 uses the worker configuration properties, with the Avro converter () and the Schema Registry .
  • connector-2 uses the Avro converter () and the Schema Registry .
  • connector-3 fails because it attempts to use the connector configuration, but does not find the Schema Registry URL configuration property. The Schema Registry URL configuration property is required for Avro, Protobuf, and JSON Schema.
  • All connectors use the worker property .

Configuration Options¶

Comma-separated list of URLs for Schema Registry instances that can be used to register or look up schemas.

  • Type: list
  • Default: “”
  • Importance: high

Specify if the Serializer should attempt to register the Schema with Schema Registry

  • Type: boolean
  • Default: true
  • Importance: medium

Maximum number of schemas to create or cache locally.

  • Type: int
  • Default: 1000
  • Importance: low

Determines how to construct the subject name under which the key schema is registered with Schema Registry. For additional information, see Schema Registry Subject Name Strategy.

Any implementation of can be specified. By default, is used as subject. Specifying an implementation of is deprecated as of and if used may have some performance degradation.

  • Type: class
  • Default: class io.confluent.kafka.serializers.subject.TopicNameStrategy
  • Importance: medium

Determines how to construct the subject name under which the value schema is registered with Schema Registry. For additional information, see Schema Registry Subject Name Strategy.

Any implementation of can be specified. By default, is used as subject. Specifying an implementation of is deprecated as of and if used may have some performance degradation.

  • Type: class
  • Default: class io.confluent.kafka.serializers.subject.TopicNameStrategy
  • Importance: medium

Specify how to pick the credentials for Basic Auth header. The supported values are URL, USER_INFO and SASL_INHERIT

  • Type: string
  • Default: “URL”
  • Importance: medium

Specify the user info for Basic Auth in the form of {username}:{password}. schema.registry.basic.auth.user.info is a deprecated alias for this configuration.

  • Type: password
  • Default: “”
  • Importance: medium

The following Schema Registry dedicated properties, configurable on the client, are available on Confluent Platform version 5.4.0 (and later). To learn more, see the information on configuring clients in Additional configurations for HTTPS.

The location of the trust store file. For example,

  • Type: string
  • Default: “”
  • Importance: medium

The password for the trust store file. If a password is not set, access to the truststore is still available but integrity checking is disabled.

  • Type: password
  • Default: “”
  • Importance: medium

The location of the key store file. This is optional for the client and can be used for two-way authentication for the client. For example, .

  • Type: string
  • Default: “”
  • Importance: medium

The store password for the key store file. This is optional for the client and only needed if is configured.

  • Type: password
  • Default: “”
  • Importance: medium

The password of the private key in the key store file. This is optional for the client.

  • Type: password
  • Default: “”
  • Importance: medium

© Copyright , Confluent, Inc. Privacy Policy | Terms & Conditions. Apache, Apache Kafka, Kafka and the Kafka logo are trademarks of the Apache Software Foundation. All other trademarks, servicemarks, and copyrights are the property of their respective owners.


Last updated on Oct 09, 2021.

Built with Sphinx using a theme provided by Read the Docs.
Sours: https://docs.confluent.io/5.5.0/schema-registry/connect.html

Schema registry connect kafka

Schema Registry

Confluent Schema Registry provides a serving layer for your metadata. It provides a RESTful interface for storing and retrieving your Avro®, JSON Schema, and Protobuf schemas. It stores a versioned history of all schemas based on a specified subject name strategy, provides multiple compatibility settings and allows evolution of schemas according to the configured compatibility settings and expanded support for these schema types. It provides serializers that plug into Apache Kafka® clients that handle schema storage and retrieval for Kafka messages that are sent in any of the supported formats.

This README includes the following sections:

Documentation

Here are a few links to Schema Registry pages in the Confluent Documentation.

Quickstart API Usage examples

The following assumes you have Kafka and an instance of the Schema Registry running using the default settings. These examples, and more, are also available at API Usage examples on docs.confluent.io.

# Register a new version of a schema under the subject "Kafka-key" $ curl -X POST -H "Content-Type: application/vnd.schemaregistry.v1+json" \ --data '{"schema": "{\"type\": \"string\"}"}' \ http://localhost:8081/subjects/Kafka-key/versions {"id":1} # Register a new version of a schema under the subject "Kafka-value" $ curl -X POST -H "Content-Type: application/vnd.schemaregistry.v1+json" \ --data '{"schema": "{\"type\": \"string\"}"}' \ http://localhost:8081/subjects/Kafka-value/versions {"id":1} # List all subjects $ curl -X GET http://localhost:8081/subjects ["Kafka-value","Kafka-key"] # List all schema versions registered under the subject "Kafka-value" $ curl -X GET http://localhost:8081/subjects/Kafka-value/versions [1] # Fetch a schema by globally unique id 1 $ curl -X GET http://localhost:8081/schemas/ids/1 {"schema":"\"string\""} # Fetch version 1 of the schema registered under subject "Kafka-value" $ curl -X GET http://localhost:8081/subjects/Kafka-value/versions/1 {"subject":"Kafka-value","version":1,"id":1,"schema":"\"string\""} # Fetch the most recently registered schema under subject "Kafka-value" $ curl -X GET http://localhost:8081/subjects/Kafka-value/versions/latest {"subject":"Kafka-value","version":1,"id":1,"schema":"\"string\""} # Delete version 3 of the schema registered under subject "Kafka-value" $ curl -X DELETE http://localhost:8081/subjects/Kafka-value/versions/3 3 # Delete all versions of the schema registered under subject "Kafka-value" $ curl -X DELETE http://localhost:8081/subjects/Kafka-value [1, 2, 3, 4, 5] # Check whether a schema has been registered under subject "Kafka-key" $ curl -X POST -H "Content-Type: application/vnd.schemaregistry.v1+json" \ --data '{"schema": "{\"type\": \"string\"}"}' \ http://localhost:8081/subjects/Kafka-key {"subject":"Kafka-key","version":1,"id":1,"schema":"\"string\""} # Test compatibility of a schema with the latest schema under subject "Kafka-value" $ curl -X POST -H "Content-Type: application/vnd.schemaregistry.v1+json" \ --data '{"schema": "{\"type\": \"string\"}"}' \ http://localhost:8081/compatibility/subjects/Kafka-value/versions/latest {"is_compatible":true} # Get top level config $ curl -X GET http://localhost:8081/config {"compatibilityLevel":"BACKWARD"} # Update compatibility requirements globally $ curl -X PUT -H "Content-Type: application/vnd.schemaregistry.v1+json" \ --data '{"compatibility": "NONE"}' \ http://localhost:8081/config {"compatibility":"NONE"} # Update compatibility requirements under the subject "Kafka-value" $ curl -X PUT -H "Content-Type: application/vnd.schemaregistry.v1+json" \ --data '{"compatibility": "BACKWARD"}' \ http://localhost:8081/config/Kafka-value {"compatibility":"BACKWARD"}

Installation

You can download prebuilt versions of the schema registry as part of the Confluent Platform. To install from source, follow the instructions in the Development section.

Deployment

The REST interface to schema registry includes a built-in Jetty server. The wrapper scripts and are the recommended method of starting and stopping the service.

Development

To build a development version, you may need a development versions of common and rest-utils. After installing these, you can build the Schema Registry with Maven.

This project uses the Google Java code style to keep code clean and consistent.

To build:

To run the unit and integration tests:

To run an instance of Schema Registry against a local Kafka cluster (using the default configuration included with Kafka):

mvn exec:java -pl :kafka-schema-registry -Dexec.args="config/schema-registry.properties"

To create a packaged version, optionally skipping the tests:

mvn package [-DskipTests]

It produces:

  • Schema registry in
  • Serde tools for avro/json/protobuf in

Each of the produced contains a directory layout similar to the packaged binary versions.

You can also produce a standalone fat JAR of schema registry using the profile:

mvn package -P standalone [-DskipTests]

This generates , which includes all the dependencies as well.

OpenAPI Spec

OpenAPI (formerly known as Swagger) specifications are built automatically using on phase.

Contribute

Thanks for helping us to make Schema Registry even better!

License

The project is licensed under the Confluent Community License, except for client libs, which is under the Apache 2.0 license. See LICENSE file in each subfolder for detailed license agreement.

Sours: https://github.com/confluentinc/schema-registry
Apache Kafka® 101: Schema Registry

On this page

The MongoDB Kafka Sink Connector setting specifies the deserialization method for data it reads from a topic. The converter can deserialize the following data formats:

Format Name

Description

AVRO

An open source serialization system that provides a compact binary format and a JSON-like API. Integrates with the Confluent Schema Registry to manage schema definitions.

JSON with Schema

JSON record structure with explicit schema information to ensure the data matches the expected format.

JSON (plain)

JSON record structure without an attached schema.

RAW JSON

Serialized as a String. The JSON structure is not managed by Kafka Connect.

Note

Even when specifying the StringConverter format, the RAW JSON mode fields must contain valid JSON to be parsed by the connector correctly.

For more information on Kafka data serialization, see Kafka Connect Serialization Explained.

The following configuration provides example settings that use the AVRO data format with a Schema Registry:

For more information on using a Schema Registry, see Schema Management.

The following configuration provides example settings that use the JSON with schema data format. The Kafka topic data must be in JSON format and contain top-level objects and .

The following configuration provides example settings that use the JSON without schema data format. The Kafka topic data must be in JSON format.

Note

Choose the appropriate data format

When you specify JSON without Schema, any JSON schema objects such as or are read explicitly rather than as a validation schema.

The following configuration provides example settings that use the RAW JSON data format.

Once the converter has deserialized the data from the Kafka topic, Kafka Connect creates a SinkRecord object.

The MongoDB Kafka Connector converts the into a which contains the key and value in BSON format. The converter determines the types using schema, if provided.

The connector supports all the core schema types listed in Schema.Type:

  • Array
  • Boolean
  • Bytes
  • Float32
  • Float64
  • Int16
  • INT32
  • INT64
  • INT8
  • MAP
  • STRING
  • STRUCT

The MongoDB Kafka Connector also supports the following AVRO logical types:

  • Decimal
  • Date
  • Time (millis/micros)
  • Timestamp (millis/micros)

For a sample AVRO schema that uses logical types, see AVRO Logical Type Example.

The converter handles schemas with nested key or value structures. The following is an example AVRO schema with nested fields:

In the above schema example, the field is a document with two sub-fields, and . It's also possible to specify those fields with dot notation:

Dot notation allows you to specify nested fields without using separate lines for the top-level document and its sub-documents.

The following AVRO schema demonstrates use of each supported logical types (, , , and ):

Note

If you compile your schema using AVRO code generation for your Kafka producer application, your logical types are mapped to the following non-standard Java classes:

Schema Type

Java Type

date

time-millis

time-micros

datetime-millis

datetime-micros

Sours: https://docs.mongodb.com/kafka-connector/current/kafka-sink-data-formats/

Now discussing:

Her hair still smelled deliciously sweet. They almost touched me, tickled my nerves. Her face reminded me of an unfinished papier-mâché mask. Only a small patch of skin on the right side was left intact.



1226 1227 1228 1229 1230