Skip to content

[WIP] Add Kafka invoke type for Go functions#3923

Open
aliok wants to merge 7 commits into
knative:mainfrom
aliok:2026-06-26-go-kafka
Open

[WIP] Add Kafka invoke type for Go functions#3923
aliok wants to merge 7 commits into
knative:mainfrom
aliok:2026-06-26-go-kafka

Conversation

@aliok

@aliok aliok commented Jun 26, 2026

Copy link
Copy Markdown
Member

WIP until knative-extensions/func-go#169 is merged

Summary

  • Add kafka as a new invoke type for Go functions
  • Add instanced and static Kafka templates (templates/go/.instanced-kafka/, templates/go/kafka/)
  • Add scaffolding that wires user functions to knative.dev/func-go/kafka
  • Add InstancedKafka and StaticKafka signature detection in pkg/scaffolding/signatures.go
  • Allow invoke: kafka in func.yaml

How it works

The Kafka middleware connects directly to Kafka brokers as a consumer group
member — no Knative Eventing or CloudEvents required. The function receives
raw kafka.Message structs (key, value, headers as []byte).

Configuration is done via environment variables (KAFKA_BROKERS,
KAFKA_TOPICS, KAFKA_CONSUMER_GROUP), which can be set as plain values,
from Kubernetes Secrets, or from ConfigMaps using the existing func.yaml
envs syntax:

envs:
- name: KAFKA_BROKERS
  value: "{{ secret:kafka-config:brokers }}"
- name: KAFKA_TOPICS
  value: "{{ configMap:kafka-config:topics }}"
- name: KAFKA_CONSUMER_GROUP
  value: "my-group"

Delivery semantics

  • At-least-once, in-order per partition
  • No retry, no deduplication, no exactly-once (planned as future work)

Dependencies

  • Requires func-go #169 to be merged and released first
  • Template go.mod files reference knative.dev/func-go v0.21.3 which will need a version bump once func-go is released with the kafka package

Limitations

  • Go only — no support for other languages yet

@knative-prow knative-prow Bot added the do-not-merge/work-in-progress 🤖 PR should not merge because it is a work in progress. label Jun 26, 2026
@knative-prow knative-prow Bot requested review from dsimansk and jrangelramos June 26, 2026 06:50
@knative-prow

knative-prow Bot commented Jun 26, 2026

Copy link
Copy Markdown

[APPROVALNOTIFIER] This PR is NOT APPROVED

This pull-request has been approved by: aliok
Once this PR has been reviewed and has the lgtm label, please assign lkingland for approval. For more information see the Code Review Process.

The full list of commands accepted by this bot can be found here.

Details Needs approval from an approver in each of these files:

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

@knative-prow-robot knative-prow-robot added the needs-rebase Cannot be merged due to conflicts with HEAD. label Jun 26, 2026
@knative-prow knative-prow Bot added the size/XL 🤖 PR changes 500-999 lines, ignoring generated files. label Jun 26, 2026
@aliok aliok force-pushed the 2026-06-26-go-kafka branch from b2688f4 to f6b4c65 Compare July 1, 2026 11:52
@knative-prow-robot knative-prow-robot removed the needs-rebase Cannot be merged due to conflicts with HEAD. label Jul 1, 2026
@aliok aliok changed the title [WIP] Add Kafka invoke type for Go functions Add Kafka invoke type for Go functions Jul 1, 2026
@knative-prow knative-prow Bot removed the do-not-merge/work-in-progress 🤖 PR should not merge because it is a work in progress. label Jul 1, 2026
@aliok aliok changed the title Add Kafka invoke type for Go functions [WIP] Add Kafka invoke type for Go functions Jul 1, 2026
@knative-prow knative-prow Bot added the do-not-merge/work-in-progress 🤖 PR should not merge because it is a work in progress. label Jul 1, 2026
@knative-prow

knative-prow Bot commented Jul 1, 2026

Copy link
Copy Markdown

@aliok: The following test failed, say /retest to rerun all failed tests or /retest-required to rerun all mandatory failed tests:

Test name Commit Details Required Rerun command
unit-tests_func_main 73a69cf link true /test unit-tests

Your PR dashboard.

Details

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes-sigs/prow repository. I understand the commands that are listed here.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

do-not-merge/work-in-progress 🤖 PR should not merge because it is a work in progress. size/XL 🤖 PR changes 500-999 lines, ignoring generated files.

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants