Troubleshooting
Check the original credentials, private network, and indexing path.
Kibana login fails
Use elastic, not kibana_system, for initial administrator login. Retrieve the original generated password from Elasticsearch's Railway Variables. Changing a variable alone does not rotate a persisted user.
For a lost administrator password, use Elastic's supported interactive reset tool in the existing container, then restore matching variables and verify authentication. Do not delete the volume. See OPERATIONS.md and Elastic password reset.
Intake rejects credentials
Use username shipper and Logstash's INPUT_PASSWORD. The producer variable LOGSTASH_PASSWORD must reference that input password, not Elasticsearch's writer password. Keep resolved values out of command arguments and logs.
Private hostname does not resolve
Run the producer in the same Railway project and environment. The private domain is not reachable from your laptop. For local contract testing, use python3 scripts/test_examples.py from the template repository.
Accepted event is missing
Select the elk-logs data view and widen the time range. Search for the original service and message. Check Logstash output errors and the dead letter queue. Invalid mapped fields can be rejected after intake acceptance.
Custom searchable fields belong inside attributes. Unknown top-level fields are retained but not indexed. Malformed JSON receives _jsonparsefailure and is not a validated event.
Intake times out
Inspect the private management endpoint :9600/_node/stats/pipelines for queue and DLQ occupancy. A full persistent queue can block intake. Use bounded producer retries and durable buffering; timeouts may occur after acceptance and retries may duplicate events.
The DLQ is bounded and does not replay automatically. Follow the isolated reader procedure in OPERATIONS.md; treat its output as sensitive data.
Startup or readiness fails
Check Elasticsearch first, then Logstash and Kibana logs. Confirm volume mounts, disk occupancy, private domain references, and actual user authentication. Readiness remains unavailable when provisioning credentials disagree with persisted users. After restart exhaustion, fix the cause and explicitly redeploy the affected service.
Preserve volumes, original secrets, and encryption keys. Never delete data to resolve a password mismatch.
Recovery fails after an upgrade
Do not downgrade Elasticsearch against upgraded data. Restore a verified compatible snapshot into a separate stack. See Upgrades.
Live demo is busy
The showcase uses shared resource and traffic limits. Wait and reload. The demo is synthetic and read-only. Use your own stack for ingestion tests.