ShardSpecs declare and configure shards managed by a Gazette consumer application. Users generally work with ShardSpecs in YAML form using the gazctl tool, in a analogous way to how Kubernetes resources are managed with kubectl.

A collection of shards often share configuration. To make them easier to work with, the gazctl tool converts to and from a hoisted common YAML map for end-user presentation. Each shard having no specific value for a configuration property inherits from the common map.

This conversion happens entirely within the tool – ShardSpecs sent to or queried from consumer member processes are full and complete individual specifications.

A common operational pattern is to define ShardSpecs that maintain a one-to-one correspondence with journal partitions of a topic, as is done with the below example. However ShardSpecs are highly configurable and a variety of other patterns are also possible. Labels can be used to attach metadata to ShardSpecs, and applications can interpret labels of the spec to drive customized processing behaviors.

Example YAML

# Create one shard for journals of examples.journalspace.yaml
# Compare to ShardSpec for full field definitions.
  # Etcd prefix into which recovery log hints are persisted.
  hint_prefix: /gazette/hints
  # Backups of verified recovery log hints, retained for disaster recovery.
  hint_backups: 2
  # Desired number of members which will live-tail the primary's recovery log.
  hot_standbys: 1
  # Upper-bound amount of time during which transactions *may* process messages
  # before beginning to commit. If input messages stall before reaching this
  # duration, the transaction may close immediately to minimize latency.
  max_txn_duration: 1s
  # Lower-bound amount of time during which a transaction *must* process messages
  # before beginning to commit. If input messages stall before reaching this
  # duration, the transaction will block.
  min_txn_duration: 0s
  # Prefix of the journal into which each shard's recovery log is recorded.
  recovery_log_prefix: examples/word-count/recovery-logs
  # Label common to all shards.
    - name: common-label
      value: common-value
  - id: shard-000
    # Label specific to this shard.
      - name: custom-label
      - journal: examples/word-count/deltas/part-000
  - id: shard-001
      - journal: examples/word-count/deltas/part-001
  - id: shard-002
      - journal: examples/word-count/deltas/part-002
  - id: shard-003
      - journal: examples/word-count/deltas/part-003

Etcd Revisions

Like JournalSpecs, ShardSpecs retrieved by the gazctl tool will include their respective Etcd modification revisions as field revision within the rendered YAML.

Consult the JournalSpecs documentation for more detail; ShardSpecs work analogously.

Deleting ShardSpecs

Like JournalSpecs, ShardSpecs may be deleted by adding a delete: true stanza to the YAML returned by gazctl and then applying it.

The field may be applied to an individual spec, or to the hoisted common map. Consult the JournalSpecs documentation for more detail.