Troubleshooting Manticore Crashes On Debian 12 Version 13.2.3

by JurnalWarga.com 62 views
Iklan Headers

Hey everyone! We're going to dive deep into a critical issue reported by a user experiencing crashes with Manticore Search version 13.2.3 on Debian 12. This article will break down the problem, analyze the logs, and discuss potential solutions and workarounds. If you're facing similar issues, you're in the right place! Let's get started and figure out how to keep Manticore running smoothly.

Understanding the Manticore Crash

Initial Bug Report

So, the user reported a crash in their Manticore setup. Here's a snippet from their report:

# Example cAug 6 18:07:22 Ubuntu searchd[2012673]: precaching table 'dns_logs_2025_05_28'
Aug 6 18:07:22 Ubuntu searchd[2012673]: --- crashed invalid query ---
Aug 6 18:07:22 Ubuntu searchd[2012673]: --- request dump end ---
Aug 6 18:07:22 Ubuntu searchd[2012673]: --- local index:
Aug 6 18:07:22 Ubuntu searchd[2012673]: Manticore 13.2.3 bf96368a6@25070806 (columnar 4.0.0 5aa8e43@25012409) (secondary 4.0.0 5aa8e43@25012409)
Aug 6 18:07:22 Ubuntu searchd[2012673]: Handling signal 11
Aug 6 18:07:22 Ubuntu searchd[2012673]: -------------- backtrace begins here ---------------
Aug 6 18:07:22 Debian searchd[2012673]: Program compiled with Clang 16.0.6
Aug 6 18:07:22 Debian searchd[2012673]: Built on Linux x86_64 (buster) (cross-compiled)
Aug 6 18:07:22 Debian searchd[2012673]: Stack bottom = 0x7f555cb83e47, thread stack size = 0x20000
Aug 6 18:07:22 Debian searchd[2012673]: Trying manual backtrace:
Aug 6 18:07:22 Debian searchd[2012673]: Something wrong with thread stack, manual backtrace may be incorrect (fp=0x20000)
Aug 6 18:07:22 Debian searchd[2012673]: Wrong stack limit or frame pointer, manual backtrace failed (fp=0x20000, stack=0x7f555cb90000, stacksize=0x20000)
Aug 6 18:07:22 Debian searchd[2012673]: Trying system backtrace:
Aug 6 18:07:22 Debian searchd[2012673]: begin of system symbols:
Aug 6 18:07:22 Debian searchd[2012673]: Trying boost backtrace:
Aug 6 18:07:22 Debian searchd[2012673]: 0# sphBacktrace(int, bool) in /usr/bin/searchd
Aug 6 18:07:22 Debian searchd[2012673]: 1# HandleCrash(int) in /usr/bin/searchd
Aug 6 18:07:22 Debian searchd[2012673]: 2# 0x00007F555D562050 in /lib/x86_64-linux-gnu/libc.so.6
Aug 6 18:07:22 Debian searchd[2012673]: 3# Threads::Strand_c::ScheduleOp(Threads::details::SchedulerOperation_t*, bool) in /usr/bin/searchd
Aug 6 18:07:22 Debian searchd[2012673]: 4# Threads::Coro::Worker_c::DoComplete(void*, Threads::details::SchedulerOperation_t*) in /usr/bin/searchd
Aug 6 18:07:22 Debian searchd[2012673]: 5# Threads::Service_t::do_run_one(CSphScopedLock<CSphMutex>&, Threads::TaskServiceThreadInfo_t&, std::atomic<bool>&) in /usr/bin/searchd
Aug 6 18:07:22 Debian searchd[2012673]: 6# Threads::Service_t::run(std::atomic<bool>&) in /usr/bin/searchd
Aug 6 18:07:22 Debian searchd[2012673]: 7# Threads::ThreadPool_c::loop(int) in /usr/bin/searchd
Aug 6 18:07:22 Debian searchd[2012673]: 8# 0x000056262E3C4927 in /usr/bin/searchd
Aug 6 18:07:22 Debian searchd[2012673]: 9# ThreadProcWrapper_fn(void*) in /usr/bin/searchd
Aug 6 18:07:22 Debian searchd[2012673]: 10# 0x00007F555D5AF134 in /lib/x86_64-linux-gnu/libc.so.6
Aug 6 18:07:22 Debian searchd[2012673]: 11# __clone in /lib/x86_64-linux-gnu/libc.so.6
Aug 6 18:07:22 Debian searchd[2012673]: -------------- backtrace ends here ---------------
Aug 6 18:07:22 Debian searchd[2012673]: Please, create a bug report in our bug tracker (https://github.com/manticoresoftware/manticore/issues)
Aug 6 18:07:22 Debian searchd[2012673]: and attach there:
Aug 6 18:07:22 Debian searchd[2012673]: a) searchd log, b) searchd binary, c) searchd symbols.
Aug 6 18:07:22 Debian searchd[2012673]: Look into the chapter 'Reporting bugs' in the manual
Aug 6 18:07:22 Debian searchd[2012673]: (https://manual.manticoresearch.com/Reporting_bugs)
Aug 6 18:07:22 Debian searchd[2012592]: Manticore 13.2.3 bf96368a6@25070806 (columnar 4.0.0 5aa8e43@25012409) (secondary 4.0.0 5aa8e43@25012409)
Aug 6 18:07:22 Debian searchd[2012592]: Copyright (c) 2001-2016, Andrew Aksyonoff
Aug 6 18:07:22 Debian searchd[2012592]: Copyright (c) 2008-2016, Sphinx Technologies Inc (http://sphinxsearch.com)
Aug 6 18:07:22 Debian searchd[2012592]: Copyright (c) 2017-2025, Manticore Software LTD (https://manticoresearch.com)
Aug 6 18:07:22 Debian systemd[1]: manticore.service: Failed to parse PID from file /run/manticore/searchd.pid: Invalid argument

The core issue seems to be a crash triggered by an invalid query while precaching the table dns_logs_2025_05_28. The log snippet indicates a signal 11, which usually means a segmentation fault—a common type of crash caused by accessing memory the program shouldn't. The backtrace provides some clues, but without access to the log files and binary, we're somewhat limited in our analysis. However, we can still explore potential causes and solutions.

Analyzing the Backtrace

Let's break down the backtrace. The backtrace, in essence, shows the chain of function calls that led to the crash. It's like retracing your steps to figure out where you went wrong. Key parts of the backtrace include:

  • Threads::Strand_c::ScheduleOp: This suggests the crash might be related to Manticore's threading or concurrency mechanisms. Threading issues are notorious for causing unpredictable crashes.
  • Threads::Coro::Worker_c::DoComplete: Coroutines are lightweight threads, further pointing to a potential concurrency issue.
  • Threads::Service_t::do_run_one and Threads::ThreadPool_c::loop: These indicate the crash is happening within Manticore's thread pool management, where tasks are executed.

The backtrace doesn't pinpoint the exact line of code, but it narrows down the area of concern to the threading and task scheduling within Manticore. This is incredibly useful for developers trying to reproduce and fix the bug.

Version and Environment

  • Manticore Search Version: 13.2.3
  • Operating System: Debian 12
  • The user also mentioned that they tried the latest development version, which is a good step to see if the issue has been addressed in more recent builds. If the crash persists even in the development version, it suggests a deeper, more fundamental problem.

Constraints

The user has a significant constraint: no direct machine access. This makes debugging much harder. We can't directly inspect logs, memory dumps, or run debugging tools on the system. This means we have to rely on the information provided and make educated guesses about the cause.

Potential Causes and Solutions

Given the information, let's explore some possible causes and what we can do about them.

1. Concurrency Issues and Race Conditions

  • Cause: Manticore, like many search engines, uses multithreading to handle queries and indexing efficiently. However, multithreading can introduce race conditions, where multiple threads access the same memory or data structures simultaneously, leading to corruption or crashes. The backtrace strongly suggests this.
  • Solution: This is tricky to solve without direct access. Typically, developers would use debugging tools to identify the exact race condition. However, some general strategies include:
    • Reviewing Query Patterns: Are there specific query patterns that might be triggering the crash? For example, very complex queries or a high volume of concurrent queries might exacerbate threading issues.
    • Configuration Tweaks: Adjusting Manticore's configuration, such as the number of threads used for indexing or searching, might alleviate the issue. This would be a trial-and-error approach.
    • Upgrading to a More Stable Version: Check if there are newer stable releases or patches that address known threading issues. While the user tried the development version, a specific stable release might have targeted fixes.

2. Memory Corruption

  • Cause: Segmentation faults (signal 11) are often caused by memory corruption. This can happen if Manticore is writing to memory it shouldn't be, or if memory is being freed incorrectly (e.g., double-free).
  • Solution: Again, direct debugging is ideal, but without it:
    • Check Indexing Process: If the crash happens during or shortly after indexing, there might be an issue with the indexing code. Consider re-indexing the data, possibly with a smaller subset first to see if the issue can be isolated.
    • Examine Table Definition: Incorrect table definitions, especially around data types, could lead to memory corruption. Review the table schema for any inconsistencies.

3. Query Complexity or Bugs in Query Processing

  • Cause: The log message crashed invalid query suggests the query itself might be the problem. There could be a bug in Manticore's query processing logic, especially when dealing with certain types of queries or data.
  • Solution:
    • Simplify Queries: Try running simpler queries to see if the crash still occurs. If simpler queries work, gradually increase the complexity to identify the problematic parts.
    • Check for Syntax Errors: Review the query syntax carefully. Even a small error can sometimes lead to unexpected behavior.
    • Test with Different Data: If possible, try running the same query on a smaller or different dataset. This can help determine if the issue is data-specific.

4. Operating System or Library Issues

  • Cause: Although less likely, the crash could be related to the underlying operating system (Debian 12) or system libraries. Compatibility issues or bugs in these components can sometimes manifest as application crashes.
  • Solution:
    • Check System Logs: Review system logs (e.g., /var/log/syslog or /var/log/messages) for any related errors or warnings.
    • Update System: Ensure the operating system and core libraries are up to date. Sometimes, updates include fixes for bugs that could cause crashes.

Steps to Take Without Direct Machine Access

Given the user's limitations, here’s a structured approach they can take:

  1. Reproduce the Issue Consistently: Try to identify the exact steps that lead to the crash. Can it be triggered by a specific query, during indexing, or under heavy load? Consistent reproduction is key.
  2. Simplify and Isolate: If a specific query causes the crash, simplify it step by step to pinpoint the exact part that's problematic. If it happens during indexing, try indexing smaller chunks of data.
  3. Review Configuration: Carefully review the Manticore configuration file (manticore.conf). Look for settings that might be related to threading, memory usage, or query processing. Document any changes made.
  4. Check Manticore Documentation and Forums: Search the Manticore documentation and community forums for similar issues and solutions. Others might have encountered the same problem.
  5. Provide Detailed Bug Report: File a detailed bug report on the Manticore GitHub repository (https://github.com/manticoresoftware/manticore/issues). Include:
    • Manticore version
    • Operating system version
    • Steps to reproduce the crash
    • Relevant parts of the configuration file
    • Queries that trigger the crash (if applicable)
    • Any other observations

Importance of Detailed Information in Bug Reporting

The user mentioned they couldn't share log files or the binary due to a lack of machine access. This highlights a critical point in bug reporting: the more information you can provide, the better. Log files, configuration files, and even memory dumps can be invaluable for developers trying to diagnose and fix issues. When reporting bugs, always aim to provide as much detail as possible, while respecting any security or privacy constraints.

Community Collaboration

Debugging complex issues like this often requires collaboration. If you're facing a similar problem, don't hesitate to reach out to the Manticore community. Sharing your experiences and insights can help others and potentially lead to a faster resolution.

Conclusion

The Manticore crash reported on Debian 12, version 13.2.3, appears to stem from potential concurrency or query processing issues. While limited machine access makes direct debugging challenging, a systematic approach involving reproduction, simplification, configuration review, and detailed bug reporting can help identify and address the problem. Remember, the key is to provide as much information as possible to the Manticore team and community. By working together, we can make Manticore Search even more robust and reliable. If you have insights or suggestions, please share them in the comments below!