28/03/2018
It appears a recurring issue is plaguing some users of the Cisco Automotive Resource (CAR) Scheduler, specifically concerning unexpected restarts occurring precisely at midnight. This phenomenon can be disruptive, potentially interrupting scheduled tasks and impacting reporting or data processing. The symptoms described point towards a complex interaction within the CAR system, with the CDR Loader service being a primary suspect. Let's delve into the details of this problem and explore potential solutions.
Understanding the CAR Scheduler and CDR Loader
The Cisco CAR Scheduler is a vital component responsible for managing and executing various tasks related to call accounting and reporting. It orchestrates the collection, processing, and distribution of call detail records (CDRs). The CDR Loader, as its name suggests, is the service responsible for ingesting and processing these CDRs. When the CDR Loader encounters issues or takes an unusually long time to complete its tasks, it can have a cascading effect on other CAR services, including the scheduler itself.
The Midnight Restart Conundrum
The observation that restarts consistently happen around midnight is a significant clue. This timing often coincides with the end of a business day and the commencement of batch processing jobs. In the context of CAR, the CDR Loader job typically initiates around 23:05 daily. The core of the problem seems to lie in the duration and success of this daily CDR Load job. Evidence suggests that this job, when it runs for an extended period, potentially up to 24 hours, ultimately fails to complete successfully. This failure to complete is then interpreted by the CAR Scheduler as a condition requiring a restart to ensure that planned jobs can continue to be processed.
Diagnosing the "Unfinished" Status
A key diagnostic step involves examining the CAR system's log events. By navigating to the system logs within CDR and searching for entries related to "CDR Load," users can observe the status of this critical job over several days. The pattern that emerges is often that the CDR Load job is reported as "in progress" for an entire 24-hour cycle. When the CAR Scheduler is subsequently restarted (due to the perceived failure), the CDR Load job is then logged as "unfinished." This cyclical behaviour strongly indicates that the job is not completing within its expected timeframe, leading to the scheduler's premature restart.
Potential Culprits: SFTP and Third-Party Loggers
While restarting all CDR services might seem like a logical first step, it appears this does not resolve the underlying issue. The focus then shifts to external factors that might be influencing the CDR Load process. A particularly strong candidate for causing these prolonged job durations is the method by which CDR data is fed to third-party call loggers. Specifically, if the CDR feed is configured to use Secure File Transfer Protocol (SFTP) to a third-party system, this could be the root cause.
The interaction between the CAR system and an external SFTP server, especially under heavy load or due to network latency, can lead to significant delays in the CDR processing. The CAR system might be waiting for acknowledgments or experiencing timeouts during the SFTP transfer, thereby extending the CDR Load job's execution time. When this delay pushes the job beyond a critical threshold, it triggers the restart mechanism.
Investigating SFTP Feed Issues
For those utilizing an SFTP feed to a third-party logger, a thorough investigation into the SFTP connection and transfer process is paramount. This could involve:
- Monitoring SFTP Server Performance: Check the responsiveness and resource utilization of the SFTP server receiving the CDR data.
- Network Latency Analysis: Measure the network latency between the CAR system and the SFTP server. High latency can significantly impact transfer times.
- File Size and Transfer Volume: Assess the volume of CDR data being transferred. Very large files or a high frequency of transfers might overwhelm the SFTP process.
- SFTP Client Configuration: Review the SFTP client settings within the CAR system for any misconfigurations or suboptimal parameters.
- Third-Party Logger Integration: Liaise with the vendor of the third-party call logger to understand their ingestion process and any potential bottlenecks on their end.
Troubleshooting Steps and Considerations
Beyond investigating the SFTP feed, here are some general troubleshooting steps to consider:
| Troubleshooting Step | Description | Potential Impact |
|---|---|---|
| Review CAR Logs in Detail | Examine all related logs, not just "CDR Load," for any other errors or warnings that coincide with the restart times. | Identifies other potential system issues. |
| Check System Resources | Ensure the server hosting the CAR system has adequate CPU, memory, and disk space. Resource exhaustion can slow down processes. | Prevents performance degradation. |
| Verify CDR Data Integrity | Although the feed to the third-party logger might appear fine, ensure the CDR data itself is not corrupted or malformed, which could slow down processing. | Ensures accurate data processing. |
| Schedule CDR Load During Off-Peak Hours | If possible, adjust the CDR Load job's schedule to a time with less system activity. | Reduces potential for conflicts. |
| Consider Alternative Data Transfer Methods | If SFTP is confirmed as the bottleneck, explore alternative, potentially more efficient, data transfer methods if supported by both systems. | Improves transfer efficiency. |
| Contact Cisco TAC | If the issue persists after thorough investigation, engaging Cisco Technical Assistance Center (TAC) is recommended. They can provide in-depth analysis and potential software patches. | Expert assistance for complex issues. |
Frequently Asked Questions
Q1: Why does my CAR Scheduler restart every night at midnight?
It's likely due to the daily CDR Load job taking too long to complete, exceeding a predefined threshold, which then triggers the CAR Scheduler to restart to ensure ongoing operations.
Q2: The CDR feed to my third-party logger seems fine. Does that mean SFTP isn't the problem?
Not necessarily. The feed to the third-party logger might be successful, but the *process* of transferring that data via SFTP from the CAR system could be where the delay occurs, impacting the CDR Load job's completion time within the CAR system itself.
Q3: What is the "unfinished" status in the CAR logs?
This status indicates that a process, in this case, the CDR Load job, did not complete its execution successfully before the CAR Scheduler was restarted.
Q4: Is there a way to change the CDR Loader job's schedule?
Depending on your CAR version and configuration, it might be possible to adjust the scheduling of the CDR Load job. Consult your CAR system's administration guide or Cisco documentation.
Q5: Should I restart the CAR services daily to prevent this?
While this might temporarily mask the issue, it's not a solution. The underlying cause of the prolonged CDR Load job needs to be identified and resolved to prevent recurring restarts.
In conclusion, the midnight restarts of the Cisco CAR Scheduler are a symptom of an underlying issue, most commonly related to the performance and duration of the daily CDR Load job. A meticulous examination of system logs and a deep dive into the SFTP transfer process to third-party loggers are crucial steps in diagnosing and resolving this persistent problem. By systematically working through these troubleshooting steps, you can aim to restore the stability and reliability of your CAR system.
If you want to read more articles similar to CAR Scheduler Midnight Restarts, you can visit the Automotive category.