ShardSpecs

ShardSpecs declare and configure shards managed by a Gazette consumer application. Users generally work with ShardSpecs in YAML form using the gazctl tool, in a analogous way to how Kubernetes resources are managed with kubectl.

A collection of shards often share configuration. To make them easier to work with, the gazctl tool converts to and from a hoisted common YAML map for end-user presentation. Each shard having no specific value for a configuration property inherits from the common map.

This conversion happens entirely within the tool – ShardSpecs sent to or queried from consumer member processes are full and complete individual specifications.

A common operational pattern is to define ShardSpecs that maintain a one-to-one correspondence with journal partitions of a topic, as is done with the below example. However ShardSpecs are highly configurable and a variety of other patterns are also possible. Labels can be used to attach metadata to ShardSpecs, and applications can interpret labels of the spec to drive customized processing behaviors.

Example YAML

# Create one shard for journals of examples.journalspace.yaml
# Compare to ShardSpec for full field definitions.
common:
  # Etcd prefix into which recovery log hints are persisted.
  hint_prefix: /gazette/hints
  # Backups of verified recovery log hints, retained for disaster recovery.
  hint_backups: 2
  # Desired number of members which will live-tail the primary's recovery log.
  hot_standbys: 1
  # Upper-bound amount of time during which transactions *may* process messages
  # before beginning to commit. If input messages stall before reaching this
  # duration, the transaction may close immediately to minimize latency.
  max_txn_duration: 1s
  # Lower-bound amount of time during which a transaction *must* process messages
  # before beginning to commit. If input messages stall before reaching this
  # duration, the transaction will block.
  min_txn_duration: 0s
  # Prefix of the journal into which each shard's recovery log is recorded.
  recovery_log_prefix: examples/word-count/recovery-logs
  # Label common to all shards.
  labels:
    - name: common-label
      value: common-value
shards:
  - id: shard-000
    # Label specific to this shard.
    labels:
      - name: custom-label
    sources:
      - journal: examples/word-count/deltas/part-000
  - id: shard-001
    sources:
      - journal: examples/word-count/deltas/part-001
  - id: shard-002
    sources:
      - journal: examples/word-count/deltas/part-002
  - id: shard-003
    sources:
      - journal: examples/word-count/deltas/part-003