Add ML Kit object detection - #27
eumaninho54 wants to merge 7 commits into
Conversation
Declares the ObjectDetector HybridObject and the useObjectDetection hook, mirroring text-recognition: stream and singleImage modes, optional multiple objects and classification, and tracking ids for live frames. MLKIT_FEATURE_KEYS already listed ObjectDetection, so only the spec, the feature module and the PluginFactory branch were missing.
Adds the domain models, the ML Kit adapter and service, and the HybridObjectDetector, all behind MLKIT_OBJECT_DETECTION. The podspec already declared the flag and the GoogleMLKit/ObjectDetection dependency, and isFeatureAvailable already answered for the feature.
Adds the domain models, the ML Kit adapter and service, and the HybridObjectDetector, split across objectDetectionEnabled and objectDetectionDisabled source sets like barcode scanning. build.gradle already carried the MLKIT_OBJECT_DETECTION build config field and the com.google.mlkit:object-detection dependency; only the source set wiring was missing.
The JS suites asserted on text recognition and barcode scanning only, so the option validation added to PluginFactory, the frame-processing hook and the static-image path were all untested. Covers the rejected option values, the singleImage default that processImageObjectDetection applies and the caller's ability to override it, argument forwarding into the worklet, and detector disposal on both success and rejection.
Adds the per-feature page the other two features already had, and lists object detection everywhere the README enumerated them: the summary, the selective install for Gradle, Podfile and the Expo plugin, the API docs index, the usage sections and the roadmap table. Documents what the types alone do not say: why stream mode is the one that yields a trackingId, that the id arrives null until tracking settles, and that bounds are in the processed image's space, so drawing them over a preview needs its rotation and scaling applied.
Wires the feature into the example app the same way as the other two: live frames in CameraView, static images in ImageScreen, and options in both zustand stores. PLUGIN_ID.OBJECT_DETECTION already existed as a roadmap entry and had no implementation behind it.
ECC Tools / Security EvidenceCommit: Security evidence gate passed (success) No security-sensitive scanner-evidence gap detected. Mode: enforce Scanned 82 changed file(s). No missing scanner-evidence signal was detected. Check publication was denied or unavailable. An app owner must enable Checks: read and write, and the installation owner must approve the updated permission. |
ECC Tools / PR Risk TaxonomyCommit: PR taxonomy review recommended (neutral) Detected 2 PR taxonomy bucket(s): CI/CD Recommendation, Cost/Token Risk. Scanned 82 changed file(s). Roadmap taxonomy buckets: CI/CD RecommendationCI, dependency, coverage, and contract signals should be routed into follow-up checks or verification work. Signals:
Paths:
Cost/Token RiskAI routing, usage, and token-budget changes should include budget or usage-limit evidence. Signals:
Paths:
Check publication was denied or unavailable. An app owner must enable Checks: read and write, and the installation owner must approve the updated permission. |
ECC Tools / Reference Set ReadinessCommit: Reference set readiness gaps detected (neutral) Reference evidence present for 1/7 areas (14%) across 82 changed file(s). This check is based on files changed in this PR. Repository-level readiness is still reported by
Check publication was denied or unavailable. An app owner must enable Checks: read and write, and the installation owner must approve the updated permission. |
ECC Tools / Hosted Promotion ReadinessCommit: Hosted promotion readiness passed (success) No hosted promotion evidence gaps detected across 82 changed file(s); 0 corpus scenarios had matching evidence. This check compares PR file changes against the evaluator/RAG promotion corpus in No evaluator corpus scenarios matched this PR. Check publication was denied or unavailable. An app owner must enable Checks: read and write, and the installation owner must approve the updated permission. |
|
Thanks for the contribution! I’ll take a proper look at the PR ASAP. I’ve been a bit busy with work lately, so I haven’t had the chance to review it yet, but I’ll get back to it as soon as I can 🙌 |
ECC Tools / Security EvidenceCommit: Security evidence gate passed (success) No security-sensitive scanner-evidence gap detected. Mode: enforce Scanned 83 changed file(s). No missing scanner-evidence signal was detected. Check publication was denied or unavailable. An app owner must enable Checks: read and write, and the installation owner must approve the updated permission. |
ECC Tools / PR Risk TaxonomyCommit: PR taxonomy review recommended (neutral) Detected 2 PR taxonomy bucket(s): CI/CD Recommendation, Cost/Token Risk. Scanned 83 changed file(s). Roadmap taxonomy buckets: CI/CD RecommendationCI, dependency, coverage, and contract signals should be routed into follow-up checks or verification work. Signals:
Paths:
Cost/Token RiskAI routing, usage, and token-budget changes should include budget or usage-limit evidence. Signals:
Paths:
Check publication was denied or unavailable. An app owner must enable Checks: read and write, and the installation owner must approve the updated permission. |
ECC Tools / Reference Set ReadinessCommit: Reference set readiness gaps detected (neutral) Reference evidence present for 1/7 areas (14%) across 83 changed file(s). This check is based on files changed in this PR. Repository-level readiness is still reported by
Check publication was denied or unavailable. An app owner must enable Checks: read and write, and the installation owner must approve the updated permission. |
ECC Tools / Hosted Promotion ReadinessCommit: Hosted promotion readiness passed (success) No hosted promotion evidence gaps detected across 83 changed file(s); 0 corpus scenarios had matching evidence. This check compares PR file changes against the evaluator/RAG promotion corpus in No evaluator corpus scenarios matched this PR. Check publication was denied or unavailable. An app owner must enable Checks: read and write, and the installation owner must approve the updated permission. |
|
Thanks for the update! The CMake fix looks good, and I also compared the implementation with the existing text recognition and barcode scanning paths — object detection is following the same patterns we already use on Android. I’m going to let CI run now and do a final check once everything is green. One small thing before we merge: could you retarget the PR from main to develop? We still have a couple of follow-up items to handle before the next release, including the Android service-instance consistency and a known ROI/orientation issue. I’d rather land the feature in develop first, finish those fixes there, and then bring everything together into main with the next release. Thanks again for the contribution 🙌 |
|
done... looks like the android pipeline broke, but apparently it’s a setup thing, not something from the branch. setup-android@v3 tries to install the tools package by default, which doesn’t exist anymore. adding packages: 'platform-tools' to that step should fix it. |
2 participants
Hey man, how's it going? first of all thanks a lot for the work you've put into developing this lib, it helped me a lot on a client's app.
Recently I had the need to also do object recognition and noticed this lib hadn't implemented it yet, so I'm opening this pull request.
I based it on the same format as
text-recognitionandbarcode-scanning, any change you want me to make I'm available to fix it.ML Kit's object detection fills a gap the other two tools don't cover, it's class-agnostic, meaning it locates prominent objects without needing a trained class for them, and in
streammode it assigns atrackingIDthat stays stable across frames. That combination is what gives me the result I'm after, keeping the box locked on the object while the camera moves.Public API
detectis synchronous on purpose, that's what lets you call it from inside the frame processor worklet.detectImageis a Promise and runs on a background queue, same as the other static image paths.Native
I followed the same three layers and reused
ImagePreprocessor,StaticImageURLResolverandremap(metadata). Every ML kit import sits behind the#if MLKIT_OBJECT_DETECTIONflag on iOS, and on Android a pair of source setsobjectDetectionEnabledandobjectDetectionDisabled, since kotlin has no preprocessor. Only one line was missing in gradle'ssourceSets, the rest of the config was already there.I left two detector instances, one for frames and one for static images, same as
HybridTextRecognizer. Instreammode it keeps tracking state between calls, and sharing the same instance between the camera thread and the background queue would break that.toBoundingBoxis a literal copy fromMLKitTextAdapter, including thetop: maxYandbottom: minY, so objects and text don't end up in different conventions inside the same lib.Tests and docs
I added coverage for
PluginFactory, for the hook and for the static image path. Suite at 104 tests passing.I created
docs/object-detection.mdin the same format as the other features and updated the readme where it lists features. I documented what the types alone don't say: onlystreamreturns atrackingId, that id comesnullon the first frames until it settles, andboundscome in the processed image's space, so whoever draws on top of the preview needs to apply its rotation and scaling.Video
ScreenRecording_09-13-2026.17-56-26_1.mov