o zi;@sdddlmZddlZddlZddlZddlZddlmZmZmZddl m Z ddl Z ddl m ZddlmZmZmZmZmZddlmZmZmZdd lmZdd lmZdd lmZdd lm Z!m"Z"m#Z#m$Z$dd l%m&Z&m'Z'm(Z(ddl)m*Z*m+Z+ddl,m-Z-m.Z.ee/Z0e dZ1ej23ej24e5dZ6dZ7dZ8dZ9dZ:dZ;ddZddddZ?dS)) getLoggerN)datetimetimezone timedelta) load_dotenv) celery_app) create_deals_table_if_not_existssanitize_col_nameload_watermarksave_watermarkDB_PATH)fetch_deals_for_dealsfetch_product_batch validate_asin) TokenManager) FUNCTION_LIST)get_seller_info_for_single_deal) load_settingscalculate_all_in_costcalculate_profit_and_margincalculate_min_listing_price)get_1yr_avg_sale_priceget_percent_discount get_trend)classify_seasonalityget_sells_period)_process_single_dealclean_numeric_valuesdealsz headers.json update_recent_deals_locki2cCs(tdddtjd}|t|d}|S)zFConverts Keepa time (minutes since 2011-01-01) to ISO 8601 UTC string.rtzinfo)minutes)rrutcr isoformat) keepa_minutes keepa_epoch dt_objectr-2/var/www/agentarbitrage/keepa_deals/simple_task.py_convert_keepa_time_to_iso/sr/cCsD|sdSt|tj}tdddtjd}||}t|dS)zIConverts an ISO 8601 UTC string to Keepa time (minutes since 2011-01-01).rr$rr%<)r fromisoformat astimezonerr(int total_seconds)iso_strr,r+deltar-r-r._convert_iso_to_keepa_time5s r7z+keepa_deals.simple_task.update_recent_deals)namec2s8tjtjj}|jttd}|j ddst ddSzt dt t d}t d}|sCt dW|t d dSt|}|t}t d |d snt d |jd W|t d dSt}|durt dW|t d dSt|}t d|d|dd}g} d} d} d} t d | tkrt dtdn&t| tkrt dt| dd} n|d st d|jdd} nd} t d| d| d}d}t|D]S}z"t| || |d\}}}|dur |||rd |vrWn1WqtyD}z"t d!| d"|d#d$|d%|t !d&|d#WYd}~qd}~ww|rMd |vri| dkrct d'W|t d dSt d(nr|d d)svt d*ned+d,|d d)D}| dkr|r|d"d-d}d}|D].}|d-|krt d.|"d/d0|d-d1|d2||d-d3 d}n| #|q|rt d4| d5n | d#7} t !d#q| st d6W|t d dSt d7t| d8t d9i}d:d,| D}tdt|t$D]L}|||t$}d;t|}|%|t&||d|vrV|rH|"d?sV|d>D] }|||d/<qLt !d@qt dAt|dBt dCt't( }t)*|} Wdn 1swYg}!| D]=}|d/}"|"|vrq||"}#|#+|t,|#||}$t-|#|$|}%|%rt.|%}%t/0t1j23|%dD<dE|%dF<|!#|%qt dGt|!dH|!st dIW|t d dSt dJt|!dKzt45t6}&|&7}'dLd,| D}(|(8dDdFgg})|!D]t9fdMdN| D"dD"dFf}*|)#|*qdO:dPdN|(D}+dO:dQgt|(},dO:dRdN|(D}-dSt;dT|+dU|,dV|-}.|'<|.|)|&=t dW|'j>dXdYd,|!D}/|/rtj?dZ|/gd[t d\t|/d]Wdn 1swYWn:t4j@y}zt jd^|dd_WYd}~n!d}~wty}zt jd`|dd_WYd}~nd}~ww| p| }0|0r||krtA|}1tB|1t da|1n | r| st dbt dcW|t d dS|t d w)dN)timeoutF)blockingzI--- Task: update_recent_deals is already running. Skipping execution. ---z)--- Task: update_recent_deals started --- KEEPA_API_KEY XAI_TOKENz KEEPA_API_KEY not set. Aborting.z=--- Task: update_recent_deals finished and lock released. ---z"Step 1: Initializing Delta Sync...zUpserter: Low tokens (z ). Skipping run to allow refill.zfCRITICAL: Watermark not found. The backfiller must be run at least once before the upserter. Aborting.zLoaded watermark: z (Keepa time: )rz4Step 2: Paginating through deals to find new ones...Tz/Safety Limit Reached: Stopped pagination after z pages to prevent runaway task.zNew Deal Limit Reached: Found zI new deals. Stopping fetch to process current batch and update watermark.zLow tokens during pagination (z). Stopping fetch loop.z Fetching deals using Sort Type: z (Last Update). Page: ) sort_type token_managerrzFetch failed on page z (Attempt r/z): zMFailed to fetch Page 0 after retries. Aborting task to prevent partial state.z5Failed to fetch subsequent page. Stopping pagination.drz6No more deals found (empty list). Stopping pagination.cSsg|] }t|dr|qSasin)rget.0dr-r-r. sz'update_recent_deals.. lastUpdatezStop Trigger: Deal rGz (Update: z) <= Watermark (z ). Diff: z min.z.Found a deal older than the watermark on page z. Stopping pagination.z7Step 2 Complete: No new deals found since the last run.zStep 2 Complete: Found z new deals.z.Step 3: Fetching product data for new deals...cSsg|]}|dqSrFr-rIr-r-r.rLiG)dayshistoryoffersproductserror_status_codez*Step 3 Complete: Fetched product data for z ASINs.zStep 4: Processing deals... last_seen_utcupsertersourcezStep 5 Complete: Processed z deals.z!No rows to upsert. Task finished.zStep 6: Upserting z rows into database...cSsg|]}t|qSr-)r rJhr-r-r.rL rNc3s|]}|VqdS)N)rHrYrow_dictr-r. sz&update_recent_deals..z, css|] }d|dVqdS)"Nr-rYr-r-r.r]s?css*|]}|dkrd|d|dVqdS)ASINr^z "=excluded."Nr-rYr-r-r.r]s(z INSERT INTO z (z ) VALUES (z") ON CONFLICT(ASIN) DO UPDATE SET z/Step 6 Complete: Successfully upserted/updated z rows.cSsg|] }d|vr|dqS)r`r-)rJrowr-r-r.rLsz4keepa_deals.sp_api_tasks.check_restriction_for_asins)argsz$--- Triggered restriction check for z new ASINs from upserter. ---z-Step 6 Failed: Database error during upsert: )exc_infoz/Step 6 Failed: Unexpected error during upsert: z"Successfully updated watermark to zXTask was incomplete due to token limits. Skipping watermark update to prevent data loss.z*--- Task: update_recent_deals finished ---)CredisRedisfrom_urlceleryconf broker_urllockLOCK_KEY LOCK_TIMEOUTacquireloggerinfor osgetenverrorreleaser sync_tokensbusiness_load_settingshas_enough_tokenswarningtokensr r7MAX_PAGES_PER_RUNlenMAX_NEW_DEALS_PER_RUNrangerupdate_after_call ExceptiontimesleeprHappendMAX_ASINS_PER_BATCHrequest_permission_for_callropen HEADERS_PATHjsonloadupdaterrrrnowrr(r)sqlite3connectr cursorextendtuplejoin TABLE_NAME executemanycommitrowcount send_taskErrorr/r )2 redis_clientrjapi_key xai_api_keyrBbusiness_settings watermark_isowatermark_keepa_timenewest_deal_timestamp all_new_dealspageincomplete_runhit_new_deal_limitSORT_TYPE_LAST_UPDATE deal_responsemax_page_retriesattempttokens_consumed tokens_lefte deals_on_pagefound_older_dealdealall_fetched_products asin_listi batch_asinsestimated_costproduct_responseapi_infopfheadersrows_to_upsertrG product_dataseller_data_cache processed_rowconnrsanitized_headersdata_for_upsert row_tuplecols_strvals_str update_str upsert_sql new_asinsshould_update_watermarknew_watermark_isor-r[r.update_recent_deals?s      _  N I      &    6 H r     "          3 *        r)@loggingrrprrrrrrdotenvrrdworkerrrgdb_utilsr r r r r keepa_apirrrrBrfield_mappingsr seller_inforbusiness_calculationsrrurrr new_analyticsrrrseasonality_classifierrr processingrr__name__rnrpathrdirname__file__rrrkrlryr{r/r7taskrr-r-r-r.s>