Milan Gress (3f142aa7) at 09 Jan 21:48
Milan Gress (90e8e871) at 20 Dec 17:36
add general batch job error message
Milan Gress (cd449341) at 20 Dec 17:25
add error states to menu
ok i just fixed the case where it would sometimes go back to "waiting for other jobs" right at the end for a split second
Milan Gress (3f142aa7) at 20 Dec 16:51
better handling of race condition
there is also a new conflict going on… Failed jobs are marked as completed = false
in batchJobs but progress.percent = 100
in $jobs -> so it gets stuck
oh, and this is also creating the race condition because currently progress.percent = 100
is considered as "not running" and if a job is not running and also batch jobs say that it is completed = false
then it thinks it is waiting for another job.
Originally reported her: #1851 (comment 364097)
I seem to get stuck at statistics
at 60% multiple times. The Asset ID where I last got stuck is http://id.motionbank.org/assets/c32997de-107c-4e66-91ab-ac5f9fba2f8d
.
I also added the responses for Jobs and batchjobs for debugging. The video is fairly short and is at 60% the last 4-5 hours. I can also look out for other files if you need?
Edit: now stuck for 20 hours, so I guess something failed?
[
{
"title": "People detection without tracking",
"type": "bounding-box",
"status": "ready",
"completed": true,
"creator": {
"name": "Milan 🫠 Gress",
"id": "http://id.motionbank.org/users/e43e0a07-91ae-4589-a9ad-c425ae7da706"
},
"batchJobId": "63a0aa126ca2b4c5cb344905",
"config": {
"path": "bounding-box"
},
"data": {
"assetsBucket": "vortanz-beta",
"asset": {
"creator": {
"name": "Milan 🫠 Gress",
"id": "http://id.motionbank.org/users/e43e0a07-91ae-4589-a9ad-c425ae7da706"
},
"filename": "pexels-anthony-shkraba-7569650.mp4",
"mimeType": "video/mp4",
"size": 28756302,
"tags": [],
"blobId": "e43e0a07-91ae-4589-a9ad-c425ae7da706/947f3b3361944a0f45aa4c2b91488151.mp4",
"_acl": [],
"created": "2022-12-02T16:44:05.344Z",
"modified": "2022-12-02T16:44:05.344Z",
"__v": 0,
"_creatorIds": [
"http://id.motionbank.org/users/e43e0a07-91ae-4589-a9ad-c425ae7da706",
"http://id.motionbank.org/users/e43e0a07-91ae-4589-a9ad-c425ae7da706"
],
"id": "http://id.motionbank.org/assets/c32997de-107c-4e66-91ab-ac5f9fba2f8d"
},
"assetUri": "http://10.200.14.86:9000/vortanz-beta/e43e0a07-91ae-4589-a9ad-c425ae7da706/947f3b3361944a0f45aa4c2b91488151-standard.mp4?X-Amz-Algorithm=AWS4-HMAC-SHA256&X-Amz-Credential=minio%2F20221219%2Fus-east-1%2Fs3%2Faws4_request&X-Amz-Date=20221219T181442Z&X-Amz-Expires=86400&X-Amz-SignedHeaders=host&X-Amz-Signature=a175144bebfdac49d91ae3e36a373d5446d0e4c0f7a47eadb05b268344a033eb",
"blobId": "e43e0a07-91ae-4589-a9ad-c425ae7da706/947f3b3361944a0f45aa4c2b91488151-standard.mp4"
},
"created": "2022-12-19T18:14:42.034Z",
"modified": "2022-12-19T18:15:20.240Z",
"__v": 0,
"progress": {
"percent": 100
},
"id": "63a0aa126ca2b4dc7b34490d"
},
{
"title": "statistics",
"type": "statistics",
"status": "processing",
"completed": false,
"creator": {
"name": "Milan 🫠 Gress",
"id": "http://id.motionbank.org/users/e43e0a07-91ae-4589-a9ad-c425ae7da706"
},
"batchJobId": "63a0aa126ca2b4c5cb344905",
"config": {
"path": "statistics"
},
"data": {
"assetsBucket": "vortanz-beta",
"asset": {
"creator": {
"name": "Milan 🫠 Gress",
"id": "http://id.motionbank.org/users/e43e0a07-91ae-4589-a9ad-c425ae7da706"
},
"filename": "pexels-anthony-shkraba-7569650.mp4",
"mimeType": "video/mp4",
"size": 28756302,
"tags": [],
"blobId": "e43e0a07-91ae-4589-a9ad-c425ae7da706/947f3b3361944a0f45aa4c2b91488151.mp4",
"_acl": [],
"created": "2022-12-02T16:44:05.344Z",
"modified": "2022-12-02T16:44:05.344Z",
"__v": 0,
"_creatorIds": [
"http://id.motionbank.org/users/e43e0a07-91ae-4589-a9ad-c425ae7da706",
"http://id.motionbank.org/users/e43e0a07-91ae-4589-a9ad-c425ae7da706"
],
"id": "http://id.motionbank.org/assets/c32997de-107c-4e66-91ab-ac5f9fba2f8d"
},
"assetUri": "http://10.200.14.86:9000/vortanz-beta/e43e0a07-91ae-4589-a9ad-c425ae7da706/947f3b3361944a0f45aa4c2b91488151-standard.mp4?X-Amz-Algorithm=AWS4-HMAC-SHA256&X-Amz-Credential=minio%2F20221219%2Fus-east-1%2Fs3%2Faws4_request&X-Amz-Date=20221219T181442Z&X-Amz-Expires=86400&X-Amz-SignedHeaders=host&X-Amz-Signature=a175144bebfdac49d91ae3e36a373d5446d0e4c0f7a47eadb05b268344a033eb",
"blobId": "e43e0a07-91ae-4589-a9ad-c425ae7da706/947f3b3361944a0f45aa4c2b91488151-standard.mp4"
},
"created": "2022-12-19T18:15:26.739Z",
"modified": "2022-12-19T18:15:30.157Z",
"__v": 0,
"progress": {
"percent": 60
},
"id": "63a0aa3e3aa2b6b75241c61c"
},
{
"title": "Tracking for people detection",
"type": "tracking",
"status": "ready",
"completed": true,
"creator": {
"name": "Milan 🫠 Gress",
"id": "http://id.motionbank.org/users/e43e0a07-91ae-4589-a9ad-c425ae7da706"
},
"batchJobId": "63a0aa126ca2b4c5cb344905",
"config": {
"path": "tracking"
},
"data": {
"assetsBucket": "vortanz-beta",
"asset": {
"creator": {
"name": "Milan 🫠 Gress",
"id": "http://id.motionbank.org/users/e43e0a07-91ae-4589-a9ad-c425ae7da706"
},
"filename": "pexels-anthony-shkraba-7569650.mp4",
"mimeType": "video/mp4",
"size": 28756302,
"tags": [],
"blobId": "e43e0a07-91ae-4589-a9ad-c425ae7da706/947f3b3361944a0f45aa4c2b91488151.mp4",
"_acl": [],
"created": "2022-12-02T16:44:05.344Z",
"modified": "2022-12-02T16:44:05.344Z",
"__v": 0,
"_creatorIds": [
"http://id.motionbank.org/users/e43e0a07-91ae-4589-a9ad-c425ae7da706",
"http://id.motionbank.org/users/e43e0a07-91ae-4589-a9ad-c425ae7da706"
],
"id": "http://id.motionbank.org/assets/c32997de-107c-4e66-91ab-ac5f9fba2f8d"
},
"assetUri": "http://10.200.14.86:9000/vortanz-beta/e43e0a07-91ae-4589-a9ad-c425ae7da706/947f3b3361944a0f45aa4c2b91488151-standard.mp4?X-Amz-Algorithm=AWS4-HMAC-SHA256&X-Amz-Credential=minio%2F20221219%2Fus-east-1%2Fs3%2Faws4_request&X-Amz-Date=20221219T181442Z&X-Amz-Expires=86400&X-Amz-SignedHeaders=host&X-Amz-Signature=a175144bebfdac49d91ae3e36a373d5446d0e4c0f7a47eadb05b268344a033eb",
"blobId": "e43e0a07-91ae-4589-a9ad-c425ae7da706/947f3b3361944a0f45aa4c2b91488151-standard.mp4"
},
"created": "2022-12-19T18:15:20.216Z",
"modified": "2022-12-19T18:15:26.760Z",
"__v": 0,
"progress": {
"percent": 100
},
"id": "63a0aa383aa2b605b241c254"
}
]
[
{
"_id": "63a0aa126ca2b4c5cb344905",
"data": {
"assetsBucket": "vortanz-beta",
"asset": {
"creator": {
"name": "Milan 🫠 Gress",
"id": "http://id.motionbank.org/users/e43e0a07-91ae-4589-a9ad-c425ae7da706"
},
"filename": "pexels-anthony-shkraba-7569650.mp4",
"mimeType": "video/mp4",
"size": 28756302,
"tags": [],
"blobId": "e43e0a07-91ae-4589-a9ad-c425ae7da706/947f3b3361944a0f45aa4c2b91488151.mp4",
"_acl": [],
"created": "2022-12-02T16:44:05.344Z",
"modified": "2022-12-02T16:44:05.344Z",
"__v": 0,
"_creatorIds": [
"http://id.motionbank.org/users/e43e0a07-91ae-4589-a9ad-c425ae7da706",
"http://id.motionbank.org/users/e43e0a07-91ae-4589-a9ad-c425ae7da706"
],
"id": "http://id.motionbank.org/assets/c32997de-107c-4e66-91ab-ac5f9fba2f8d"
},
"assetUri": "http://10.200.14.86:9000/vortanz-beta/e43e0a07-91ae-4589-a9ad-c425ae7da706/947f3b3361944a0f45aa4c2b91488151-standard.mp4?X-Amz-Algorithm=AWS4-HMAC-SHA256&X-Amz-Credential=minio%2F20221219%2Fus-east-1%2Fs3%2Faws4_request&X-Amz-Date=20221219T181442Z&X-Amz-Expires=86400&X-Amz-SignedHeaders=host&X-Amz-Signature=a175144bebfdac49d91ae3e36a373d5446d0e4c0f7a47eadb05b268344a033eb",
"blobId": "e43e0a07-91ae-4589-a9ad-c425ae7da706/947f3b3361944a0f45aa4c2b91488151-standard.mp4"
},
"creator": {
"name": "Milan 🫠 Gress",
"id": "http://id.motionbank.org/users/e43e0a07-91ae-4589-a9ad-c425ae7da706"
},
"jobs": [
{
"completed": true,
"_id": "63a0aa126ca2b4fa8d344906",
"path": "bounding-box",
"id": "63a0aa126ca2b4dc7b34490d"
},
{
"completed": true,
"_id": "63a0aa126ca2b49b04344907",
"path": "tracking",
"id": "63a0aa383aa2b605b241c254"
},
{
"completed": false,
"_id": "63a0aa126ca2b4e15f344908",
"path": "statistics",
"id": "63a0aa3e3aa2b6b75241c61c"
},
{
"completed": false,
"_id": "63a0aa126ca2b4f402344909",
"path": "pose"
},
{
"completed": false,
"_id": "63a0aa126ca2b4427c34490a",
"path": "har"
}
],
"created": "2022-12-19T18:14:42.024Z",
"modified": "2022-12-19T18:15:26.757Z",
"__v": 0
}
]
Ok, I changed the wording and fixed the disabled pose button
Closes #1851
Multiple problems:
currently, if there is a significant pause between the jobs, (but a batch job is still running) the polling stops because it thinks everything is done
There are two implementations of the polling (magic menu and files) which I find annoying, so probably it should move to the store? Or at the very least abstracted away into a general function.
Currently, it is implemented as a recursive function (seemed sensible at the time) but in fact if it is called from multiple sources then obviously multiple polling call stacks are created so likely a setInterval()
with a clean start/stop interface makes more sense.
Oh and there should probably a very slow default polling mode for even if it thinks nothing happens/runs to avoid getting stuck in dysfunctional state.
Milan Gress (4f3cda88) at 20 Dec 15:10
styling
Thanks for the write-up!
pose recognition
on top and in the batch jobs. The magic menu has to wait that both say the same thing.completed=true
but there are places where this is not implemented → results in flickering or falling back to step "no progess" (with you saw… i guess)So there is a problem right now with the indication of "preparing" maybe it could be something like:
Queued: pose tracking
waiting for: pose recognition
I feel like "preparing" is appropriate for the state of "no progress" but perhaps we need a better name for "future" jobs?
Or we could use "XY in progress" for both "future" & "no progress" jobs, but mark "future" jobs with "waiting for other jobs to complete" as you suggested? Also, it is entirely possible that the whole chain is skipped, and we go from "future" job to "done" without going through all the steps… this is entirely dependent on the API and how often we get new information from the API… but you already saw that
To the last image: It could also be that these are, in fact, different process? Probably not but I added the IDs of the processes in the file menu, so we can treat it more like a debug menu if stuff like this occurs (we can remove that later, but I think for now that's useful)
@anton.koch don't know if it should be its own issue, but somehow I seem to get stuck at statistics
at 60% multiple times. The Asset ID where I last got stuck is http://id.motionbank.org/assets/c32997de-107c-4e66-91ab-ac5f9fba2f8d
.
I also added the responses for Jobs and batchjobs for debugging. The video is fairly short and is at 60% the last 4-5 hours. I can also look out for other files if you need?
[
{
"title": "People detection without tracking",
"type": "bounding-box",
"status": "ready",
"completed": true,
"creator": {
"name": "Milan 🫠 Gress",
"id": "http://id.motionbank.org/users/e43e0a07-91ae-4589-a9ad-c425ae7da706"
},
"batchJobId": "63a0aa126ca2b4c5cb344905",
"config": {
"path": "bounding-box"
},
"data": {
"assetsBucket": "vortanz-beta",
"asset": {
"creator": {
"name": "Milan 🫠 Gress",
"id": "http://id.motionbank.org/users/e43e0a07-91ae-4589-a9ad-c425ae7da706"
},
"filename": "pexels-anthony-shkraba-7569650.mp4",
"mimeType": "video/mp4",
"size": 28756302,
"tags": [],
"blobId": "e43e0a07-91ae-4589-a9ad-c425ae7da706/947f3b3361944a0f45aa4c2b91488151.mp4",
"_acl": [],
"created": "2022-12-02T16:44:05.344Z",
"modified": "2022-12-02T16:44:05.344Z",
"__v": 0,
"_creatorIds": [
"http://id.motionbank.org/users/e43e0a07-91ae-4589-a9ad-c425ae7da706",
"http://id.motionbank.org/users/e43e0a07-91ae-4589-a9ad-c425ae7da706"
],
"id": "http://id.motionbank.org/assets/c32997de-107c-4e66-91ab-ac5f9fba2f8d"
},
"assetUri": "http://10.200.14.86:9000/vortanz-beta/e43e0a07-91ae-4589-a9ad-c425ae7da706/947f3b3361944a0f45aa4c2b91488151-standard.mp4?X-Amz-Algorithm=AWS4-HMAC-SHA256&X-Amz-Credential=minio%2F20221219%2Fus-east-1%2Fs3%2Faws4_request&X-Amz-Date=20221219T181442Z&X-Amz-Expires=86400&X-Amz-SignedHeaders=host&X-Amz-Signature=a175144bebfdac49d91ae3e36a373d5446d0e4c0f7a47eadb05b268344a033eb",
"blobId": "e43e0a07-91ae-4589-a9ad-c425ae7da706/947f3b3361944a0f45aa4c2b91488151-standard.mp4"
},
"created": "2022-12-19T18:14:42.034Z",
"modified": "2022-12-19T18:15:20.240Z",
"__v": 0,
"progress": {
"percent": 100
},
"id": "63a0aa126ca2b4dc7b34490d"
},
{
"title": "statistics",
"type": "statistics",
"status": "processing",
"completed": false,
"creator": {
"name": "Milan 🫠 Gress",
"id": "http://id.motionbank.org/users/e43e0a07-91ae-4589-a9ad-c425ae7da706"
},
"batchJobId": "63a0aa126ca2b4c5cb344905",
"config": {
"path": "statistics"
},
"data": {
"assetsBucket": "vortanz-beta",
"asset": {
"creator": {
"name": "Milan 🫠 Gress",
"id": "http://id.motionbank.org/users/e43e0a07-91ae-4589-a9ad-c425ae7da706"
},
"filename": "pexels-anthony-shkraba-7569650.mp4",
"mimeType": "video/mp4",
"size": 28756302,
"tags": [],
"blobId": "e43e0a07-91ae-4589-a9ad-c425ae7da706/947f3b3361944a0f45aa4c2b91488151.mp4",
"_acl": [],
"created": "2022-12-02T16:44:05.344Z",
"modified": "2022-12-02T16:44:05.344Z",
"__v": 0,
"_creatorIds": [
"http://id.motionbank.org/users/e43e0a07-91ae-4589-a9ad-c425ae7da706",
"http://id.motionbank.org/users/e43e0a07-91ae-4589-a9ad-c425ae7da706"
],
"id": "http://id.motionbank.org/assets/c32997de-107c-4e66-91ab-ac5f9fba2f8d"
},
"assetUri": "http://10.200.14.86:9000/vortanz-beta/e43e0a07-91ae-4589-a9ad-c425ae7da706/947f3b3361944a0f45aa4c2b91488151-standard.mp4?X-Amz-Algorithm=AWS4-HMAC-SHA256&X-Amz-Credential=minio%2F20221219%2Fus-east-1%2Fs3%2Faws4_request&X-Amz-Date=20221219T181442Z&X-Amz-Expires=86400&X-Amz-SignedHeaders=host&X-Amz-Signature=a175144bebfdac49d91ae3e36a373d5446d0e4c0f7a47eadb05b268344a033eb",
"blobId": "e43e0a07-91ae-4589-a9ad-c425ae7da706/947f3b3361944a0f45aa4c2b91488151-standard.mp4"
},
"created": "2022-12-19T18:15:26.739Z",
"modified": "2022-12-19T18:15:30.157Z",
"__v": 0,
"progress": {
"percent": 60
},
"id": "63a0aa3e3aa2b6b75241c61c"
},
{
"title": "Tracking for people detection",
"type": "tracking",
"status": "ready",
"completed": true,
"creator": {
"name": "Milan 🫠 Gress",
"id": "http://id.motionbank.org/users/e43e0a07-91ae-4589-a9ad-c425ae7da706"
},
"batchJobId": "63a0aa126ca2b4c5cb344905",
"config": {
"path": "tracking"
},
"data": {
"assetsBucket": "vortanz-beta",
"asset": {
"creator": {
"name": "Milan 🫠 Gress",
"id": "http://id.motionbank.org/users/e43e0a07-91ae-4589-a9ad-c425ae7da706"
},
"filename": "pexels-anthony-shkraba-7569650.mp4",
"mimeType": "video/mp4",
"size": 28756302,
"tags": [],
"blobId": "e43e0a07-91ae-4589-a9ad-c425ae7da706/947f3b3361944a0f45aa4c2b91488151.mp4",
"_acl": [],
"created": "2022-12-02T16:44:05.344Z",
"modified": "2022-12-02T16:44:05.344Z",
"__v": 0,
"_creatorIds": [
"http://id.motionbank.org/users/e43e0a07-91ae-4589-a9ad-c425ae7da706",
"http://id.motionbank.org/users/e43e0a07-91ae-4589-a9ad-c425ae7da706"
],
"id": "http://id.motionbank.org/assets/c32997de-107c-4e66-91ab-ac5f9fba2f8d"
},
"assetUri": "http://10.200.14.86:9000/vortanz-beta/e43e0a07-91ae-4589-a9ad-c425ae7da706/947f3b3361944a0f45aa4c2b91488151-standard.mp4?X-Amz-Algorithm=AWS4-HMAC-SHA256&X-Amz-Credential=minio%2F20221219%2Fus-east-1%2Fs3%2Faws4_request&X-Amz-Date=20221219T181442Z&X-Amz-Expires=86400&X-Amz-SignedHeaders=host&X-Amz-Signature=a175144bebfdac49d91ae3e36a373d5446d0e4c0f7a47eadb05b268344a033eb",
"blobId": "e43e0a07-91ae-4589-a9ad-c425ae7da706/947f3b3361944a0f45aa4c2b91488151-standard.mp4"
},
"created": "2022-12-19T18:15:20.216Z",
"modified": "2022-12-19T18:15:26.760Z",
"__v": 0,
"progress": {
"percent": 100
},
"id": "63a0aa383aa2b605b241c254"
}
]
[
{
"_id": "63a0aa126ca2b4c5cb344905",
"data": {
"assetsBucket": "vortanz-beta",
"asset": {
"creator": {
"name": "Milan 🫠 Gress",
"id": "http://id.motionbank.org/users/e43e0a07-91ae-4589-a9ad-c425ae7da706"
},
"filename": "pexels-anthony-shkraba-7569650.mp4",
"mimeType": "video/mp4",
"size": 28756302,
"tags": [],
"blobId": "e43e0a07-91ae-4589-a9ad-c425ae7da706/947f3b3361944a0f45aa4c2b91488151.mp4",
"_acl": [],
"created": "2022-12-02T16:44:05.344Z",
"modified": "2022-12-02T16:44:05.344Z",
"__v": 0,
"_creatorIds": [
"http://id.motionbank.org/users/e43e0a07-91ae-4589-a9ad-c425ae7da706",
"http://id.motionbank.org/users/e43e0a07-91ae-4589-a9ad-c425ae7da706"
],
"id": "http://id.motionbank.org/assets/c32997de-107c-4e66-91ab-ac5f9fba2f8d"
},
"assetUri": "http://10.200.14.86:9000/vortanz-beta/e43e0a07-91ae-4589-a9ad-c425ae7da706/947f3b3361944a0f45aa4c2b91488151-standard.mp4?X-Amz-Algorithm=AWS4-HMAC-SHA256&X-Amz-Credential=minio%2F20221219%2Fus-east-1%2Fs3%2Faws4_request&X-Amz-Date=20221219T181442Z&X-Amz-Expires=86400&X-Amz-SignedHeaders=host&X-Amz-Signature=a175144bebfdac49d91ae3e36a373d5446d0e4c0f7a47eadb05b268344a033eb",
"blobId": "e43e0a07-91ae-4589-a9ad-c425ae7da706/947f3b3361944a0f45aa4c2b91488151-standard.mp4"
},
"creator": {
"name": "Milan 🫠 Gress",
"id": "http://id.motionbank.org/users/e43e0a07-91ae-4589-a9ad-c425ae7da706"
},
"jobs": [
{
"completed": true,
"_id": "63a0aa126ca2b4fa8d344906",
"path": "bounding-box",
"id": "63a0aa126ca2b4dc7b34490d"
},
{
"completed": true,
"_id": "63a0aa126ca2b49b04344907",
"path": "tracking",
"id": "63a0aa383aa2b605b241c254"
},
{
"completed": false,
"_id": "63a0aa126ca2b4e15f344908",
"path": "statistics",
"id": "63a0aa3e3aa2b6b75241c61c"
},
{
"completed": false,
"_id": "63a0aa126ca2b4f402344909",
"path": "pose"
},
{
"completed": false,
"_id": "63a0aa126ca2b4427c34490a",
"path": "har"
}
],
"created": "2022-12-19T18:14:42.024Z",
"modified": "2022-12-19T18:15:26.757Z",
"__v": 0
}
]