Skip to content

Commit b050b1b

Browse files
wileyjwileyj
authored
chore: update rosetta dockerfile (#1651)
* Rewrite of rosetta dockerfile This close to a full rewrite of the built image. - runit is removed - `/entrypoint.sh` now starts all processes via single script - all logs are piped to stdout - remove printf in favor of heredoc - add a script to seed the chainstate when env var `SEED_CHAINSTATE=true` - adds a postgres init script in case of genesis sync and DB != `postgres` * chore - address PR comments updating docs where stx-rosetta is used add some docs around `SEED_CHAINSTATE` add -W to pg_ctl and removing sleep
1 parent 2015b9c commit b050b1b

File tree

5 files changed

+354
-163
lines changed

5 files changed

+354
-163
lines changed

‎.github/workflows/ci.yml

+1-1
Original file line numberDiff line numberDiff line change
@@ -955,7 +955,7 @@ jobs:
955955
context: .
956956
build-args: |
957957
STACKS_API_VERSION=${{ github.head_ref || github.ref_name }}
958-
file: docker/stx-rosetta.Dockerfile
958+
file: docker/rosetta.Dockerfile
959959
tags: ${{ steps.meta_standalone.outputs.tags }}
960960
labels: ${{ steps.meta_standalone.outputs.labels }}
961961
# Only push if (there's a new release on main branch, or if building a non-main branch) and (Only run on non-PR events or only PRs that aren't from forks)

‎README-rosetta.md

+10-4
Original file line numberDiff line numberDiff line change
@@ -1,16 +1,22 @@
11
# Testing the Rosetta APIs
22

3-
Build and run the `stx-rosetta.Dockerfile` image:
3+
Build and run the `rosetta.Dockerfile` image:
44

5-
docker build -t stx-rosetta:stable -f stx-rosetta.Dockerfile .
5+
docker build -t rosetta:stable -f rosetta.Dockerfile .
66
docker run -d -p 3999:3999 --mount source=rosetta-data,target=/data \
7-
--name stx-rosetta stx-rosetta:stable
7+
--name rosetta rosetta:stable
88

99
By default, this will connect to the testnet. To run a local node, run
1010

1111

1212
docker run -d -p 3999:3999 --mount source=rosetta-data,target=/data \
13-
--name stx-rosetta -e STACKS_NETWORK=mocknet stx-rosetta:stable
13+
--name rosetta -e STACKS_NETWORK=mocknet rosetta:stable
14+
15+
Optionally, you can seed the chainstate for testnet/mainnet using [Hiro archive data](https://docs.hiro.so/references/hiro-archive#what-is-the-hiro-archive):
16+
17+
18+
docker run -d -p 3999:3999 --mount source=rosetta-data,target=/data \
19+
--name rosetta -e SEED_CHAINSTATE=true rosetta:stable
1420

1521
Use a recent version of [rosetta-cli](https://github.com/coinbase/rosetta-cli) to test the endpoints:
1622

‎content/feature-guides/rosetta-support.md

+44-12
Original file line numberDiff line numberDiff line change
@@ -8,19 +8,51 @@ The Stacks Blockchain API supports [v1.4.6 of the Rosetta specification](https:/
88

99
# Testing the Rosetta APIs
1010

11-
To build and run the `stx-rosetta.Dockerfile` image, run the following command:
12-
13-
```docker build -t stx-rosetta:stable -f stx-rosetta.Dockerfile .
14-
docker run -d -p 3999:3999 --mount source=rosetta-data,target=/data \
15-
--name stx-rosetta stx-rosetta:stable```
16-
By default, this will connect to the testnet. To run a local node, run the following command:
17-
```docker run -d -p 3999:3999 --mount source=rosetta-data,target=/data \
18-
--name stx-rosetta -e STACKS_NETWORK=mocknet stx-rosetta:stable```
11+
To build and run the `rosetta.Dockerfile` image, run the following command:
12+
13+
```
14+
docker build -t rosetta:stable -f rosetta.Dockerfile .
15+
docker run -d \
16+
-p 3999:3999 \
17+
--mount source=rosetta-data,target=/data \
18+
--name rosetta \
19+
rosetta:stable
20+
```
21+
22+
To build and run the `rosetta.Dockerfile` image using an [archived chainstate](https://docs.hiro.so/references/hiro-archive#what-is-the-hiro-archive), run the following command:
23+
24+
```
25+
docker build -t rosetta:stable -f rosetta.Dockerfile .
26+
docker run -d \
27+
-p 3999:3999 \
28+
-e SEED_CHAINSTATE=true \
29+
--mount source=rosetta-data,target=/data \
30+
--name rosetta \
31+
rosetta:stable
32+
```
33+
34+
35+
By default, this will connect to the mainnet. To run a local node, run the following command:
36+
37+
```
38+
docker run -d \
39+
-p 3999:3999 \
40+
-e STACKS_NETWORK=mocknet \
41+
--mount source=rosetta-data,target=/data \
42+
--name rosetta \
43+
rosetta:stable
44+
```
45+
1946
To use a recent version of [rosetta-cli](https://github.com/coinbase/rosetta-cli) to test the endpoints, use the following command:
20-
```rosetta-cli --configuration-file rosetta-cli-config/rosetta-config.json \
21-
view:block 1
22-
rosetta-cli --configuration-file rosetta-cli-config/rosetta-config.json \
23-
check:data```
47+
```
48+
rosetta-cli \
49+
--configuration-file rosetta-cli-config/rosetta-config.json \
50+
view:block 1
51+
rosetta-cli \
52+
--configuration-file rosetta-cli-config/rosetta-config.json \
53+
check:data
54+
```
55+
2456
`rosetta-cli` will then sync with the blockchain until it reaches the tip, and then exit, displaying the test results.
2557
Currently, account reconciliation is disabled; proper testing of that feature requires token transfer transactions while `rosetta-cli` is running.
2658
Documentation for the Rosetta APIs can be found [here](https://hirosystems.github.io/stacks-blockchain-api/)

‎docker/rosetta.Dockerfile

+299
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,299 @@
1+
# Running with `SEED_CHAINSTATE=true` will require the container to be launched with `--shm-size=xxxxMB` where at least 256MB is recommended
2+
ARG STACKS_API_VERSION=v7.1.10
3+
ARG STACKS_BLOCKCHAIN_VERSION=2.3.0.0.2
4+
ARG PG_VERSION=15
5+
ARG STACKS_NETWORK=mainnet
6+
ARG PG_HOST=127.0.0.1
7+
ARG PG_PORT=5432
8+
ARG PG_USER=postgres
9+
ARG PG_PASSWORD=postgres
10+
ARG SEED_CHAINSTATE=false
11+
ARG ARCHIVE_VERSION=latest
12+
13+
#######################################################################
14+
## Build the stacks-blockchain-api
15+
FROM node:16-buster as stacks-blockchain-api-build
16+
ARG STACKS_API_VERSION
17+
ENV STACKS_API_REPO=hirosystems/stacks-blockchain-api
18+
ENV STACKS_API_VERSION=${STACKS_API_VERSION}
19+
ENV DEBIAN_FRONTEND noninteractive
20+
WORKDIR /app
21+
RUN apt-get update -y \
22+
&& apt-get install -y \
23+
curl \
24+
jq \
25+
openjdk-11-jre-headless \
26+
cmake \
27+
&& git clone -b ${STACKS_API_VERSION} https://github.com/${STACKS_API_REPO} . \
28+
&& echo "GIT_TAG=$(git tag --points-at HEAD)" >> .env \
29+
&& npm config set unsafe-perm true \
30+
&& npm ci \
31+
&& npm run build \
32+
&& npm prune --production
33+
34+
#######################################################################
35+
## Build the stacks-blockchain
36+
FROM rust:buster as stacks-blockchain-build
37+
ARG STACKS_BLOCKCHAIN_VERSION
38+
ENV STACKS_NODE_REPO=stacks-network/stacks-blockchain
39+
ENV STACKS_BLOCKCHAIN_VERSION=${STACKS_BLOCKCHAIN_VERSION}
40+
ENV DEBIAN_FRONTEND noninteractive
41+
WORKDIR /src
42+
RUN apt-get update -y \
43+
&& apt-get install -y \
44+
curl \
45+
&& mkdir -p /out \
46+
&& git clone -b ${STACKS_BLOCKCHAIN_VERSION} --depth 1 https://github.com/${STACKS_NODE_REPO} . \
47+
&& cd testnet/stacks-node \
48+
&& cargo build --features monitoring_prom,slog_json --release \
49+
&& cp /src/target/release/stacks-node /out
50+
51+
#######################################################################
52+
## Build the final image with all components from build stages
53+
FROM debian:buster
54+
ARG STACKS_NETWORK
55+
ARG PG_HOST
56+
ARG PG_PORT
57+
ARG PG_USER
58+
ARG PG_PASSWORD
59+
ARG PG_VERSION
60+
ARG SEED_CHAINSTATE
61+
ARG STACKS_API_VERSION
62+
ARG STACKS_BLOCKCHAIN_VERSION
63+
ARG ARCHIVE_VERSION
64+
ENV SEED_CHAINSTATE=${SEED_CHAINSTATE}
65+
ENV STACKS_API_VERSION=${STACKS_API_VERSION}
66+
ENV STACKS_BLOCKCHAIN_VERSION=${STACKS_BLOCKCHAIN_VERSION}
67+
ENV PG_VERSION=${PG_VERSION}
68+
ENV PG_HOST=${PG_HOST}
69+
ENV PG_PORT=${PG_PORT}
70+
ENV PG_USER=${PG_USER}
71+
ENV PG_PASSWORD=${PG_PASSWORD}
72+
ENV PG_DATABASE=stacks_blockchain_api
73+
ENV PGDATA=/postgres
74+
ENV PG_SCHEMA=stacks_blockchain_api
75+
ENV STACKS_BLOCKCHAIN_DIR=/stacks-blockchain
76+
ENV STACKS_BLOCKCHAIN_API_DIR=/stacks-blockchain-api
77+
ENV STACKS_NETWORK=${STACKS_NETWORK}
78+
ENV STACKS_CORE_EVENT_PORT=3700
79+
ENV STACKS_CORE_EVENT_HOST=127.0.0.1
80+
ENV STACKS_EVENT_OBSERVER=127.0.0.1:3700
81+
ENV STACKS_BLOCKCHAIN_API_PORT=3999
82+
ENV STACKS_BLOCKCHAIN_API_HOST=0.0.0.0
83+
ENV STACKS_CORE_RPC_HOST=127.0.0.1
84+
ENV STACKS_CORE_RPC_PORT=20443
85+
ENV STACKS_CORE_P2P_PORT=20444
86+
ENV ARCHIVE_VERSION=${ARCHIVE_VERSION}
87+
ENV LANG en_US.UTF-8
88+
ENV LANGUAGE en_US:en
89+
ENV LC_ALL en_US.UTF-8
90+
ENV DEBIAN_FRONTEND noninteractive
91+
92+
RUN apt-get update \
93+
&& apt-get install -y \
94+
gnupg2 \
95+
lsb-release \
96+
curl \
97+
jq \
98+
procps \
99+
netcat \
100+
gosu \
101+
locales
102+
RUN sed -i '/en_US.UTF-8/s/^# //g' /etc/locale.gen && locale-gen
103+
104+
RUN curl -sL https://www.postgresql.org/media/keys/ACCC4CF8.asc | apt-key add - \
105+
&& echo "deb http://apt.postgresql.org/pub/repos/apt/ `lsb_release -cs`-pgdg main" > /etc/apt/sources.list.d/pgsql.list \
106+
&& curl -sL https://deb.nodesource.com/setup_16.x | bash -
107+
RUN apt-get update \
108+
&& apt-get install -y \
109+
postgresql-${PG_VERSION} \
110+
postgresql-client-${PG_VERSION} \
111+
nodejs \
112+
&& echo 'debconf debconf/frontend select Noninteractive' | debconf-set-selections
113+
RUN mkdir -p \
114+
${STACKS_BLOCKCHAIN_DIR}/data \
115+
/scripts \
116+
&& apt-get clean \
117+
&& rm -rf /var/cache/apt/* /var/lib/apt/lists/* /tmp/*
118+
COPY --from=stacks-blockchain-build /out ${STACKS_BLOCKCHAIN_DIR}
119+
COPY --from=stacks-blockchain-api-build /app ${STACKS_BLOCKCHAIN_API_DIR}
120+
COPY --from=stacks-blockchain-build /src/testnet/stacks-node/conf/*follower-conf.toml ${STACKS_BLOCKCHAIN_DIR}/
121+
122+
###################################
123+
## entrypoint.sh
124+
RUN <<EOF
125+
cat > /entrypoint.sh <<'EOM'
126+
#!/bin/bash -e
127+
exec 2>&1
128+
# enable json logging for stacks-blockchain
129+
export STACKS_LOG_JSON=1
130+
# configure postgres and start it
131+
mkdir -p "${PGDATA}" || exit 1
132+
chown -R postgres:postgres "${PGDATA}" || exit 1
133+
gosu postgres /usr/lib/postgresql/${PG_VERSION}/bin/initdb -D "${PGDATA}" --wal-segsize=512 || exit 1
134+
echo "host all all all trust" >> "$PGDATA/pg_hba.conf" || exit 1
135+
gosu postgres /usr/lib/postgresql/${PG_VERSION}/bin/pg_ctl start -w -D ${PGDATA} -o "-c listen_addresses='*'" || exit 1
136+
137+
# download archive files if flag is true
138+
if [ "${SEED_CHAINSTATE}" = "true" ]; then
139+
/scripts/seed-chainstate.sh || exit 1
140+
fi
141+
# create DB/schema if using other than default 'postgres'
142+
if [[ "${SEED_CHAINSTATE}" = "false" && "${PG_DATABASE}" != "postgres" ]]; then
143+
/scripts/postgres-initdb.sh || exit 1
144+
fi
145+
# set chain_id based on network
146+
case "${STACKS_NETWORK}" in
147+
testnet)
148+
export STACKS_CHAIN_ID=0x80000000
149+
;;
150+
*)
151+
export STACKS_CHAIN_ID=0x00000001
152+
;;
153+
esac
154+
# start stacks-blockchain and store pid
155+
${STACKS_BLOCKCHAIN_DIR}/stacks-node start --config=${STACKS_BLOCKCHAIN_DIR}/${STACKS_NETWORK}-follower-conf.toml 2>&1 &
156+
STACKS_BLOCKCHAIN_PID=$!
157+
158+
# start stacks-blockchain-api and store pid
159+
pushd /stacks-blockchain-api
160+
node ./lib/index.js 2>&1 &
161+
STACKS_API_PID=$!
162+
163+
# try to stop processes gracefully
164+
function cleanup() {
165+
echo "Exiting, signal: $1"
166+
kill $STACKS_PID 2>/dev/null && echo "stacks-blockchain exiting.."
167+
wait $STACKS_PID 2>/dev/null && echo "stacks-blockchain exited"
168+
kill $API_PID 2>/dev/null && echo "stacks-blockchain-api exiting.."
169+
wait $API_PID 2>/dev/null && echo "stacks-blockchain-api exited"
170+
echo "Postgres exiting.."
171+
gosu postgres /usr/lib/postgresql/${PG_VERSION}/bin/pg_ctl stop -W -D "$PGDATA" 2>/dev/null && echo "Postgres exited"
172+
}
173+
trap "cleanup SIGTERM" SIGTERM
174+
trap "cleanup SIGINT" SIGINT
175+
trap "cleanup SIGHUP" SIGHUP
176+
trap "cleanup EXIT" EXIT
177+
wait
178+
EOM
179+
chmod +x /entrypoint.sh
180+
EOF
181+
182+
###################################
183+
## /scripts/postgres-initdb.sh
184+
RUN <<EOF
185+
cat > /scripts/postgres-initdb.sh <<'EOM'
186+
#!/bin/bash -e
187+
# connect to postgres and create DB/schema as defined in env vars
188+
psql -v ON_ERROR_STOP=1 --username "${PG_USER}" --dbname "template1" <<-EOSQL
189+
SELECT 'CREATE DATABASE ${PG_DATABASE}' WHERE NOT EXISTS (SELECT FROM pg_database WHERE datname = '${PG_DATABASE}')\gexec
190+
\c ${PG_DATABASE};
191+
CREATE SCHEMA IF NOT EXISTS ${PG_SCHEMA};
192+
GRANT ALL PRIVILEGES ON DATABASE ${PG_DATABASE} TO ${PG_USER};
193+
EOSQL
194+
exit 0
195+
EOM
196+
chmod +x /scripts/postgres-initdb.sh
197+
EOF
198+
199+
###################################
200+
## /scripts/seed-chainstate.sh
201+
RUN <<EOF
202+
cat > /scripts/seed-chainstate.sh <<'EOM'
203+
#!/bin/bash -e
204+
exec 2>&1
205+
echo "Seeding chainstate from https://archive.hiro.so"
206+
# remove the "v" in front of the API version since the archive files do not use this naming structure
207+
LOCAL_STACKS_API_VERSION=$(echo "${STACKS_API_VERSION:1}")
208+
209+
# define URL's to download
210+
PGDUMP_URL="https://archive.hiro.so/${STACKS_NETWORK}/stacks-blockchain-api-pg/stacks-blockchain-api-pg-${PG_VERSION}-${LOCAL_STACKS_API_VERSION}-${ARCHIVE_VERSION}.dump"
211+
PGDUMP_URL_SHA256="https://archive.hiro.so/${STACKS_NETWORK}/stacks-blockchain-api-pg/stacks-blockchain-api-pg-${PG_VERSION}-${LOCAL_STACKS_API_VERSION}-${ARCHIVE_VERSION}.sha256"
212+
CHAINDATA_URL="https://archive.hiro.so/${STACKS_NETWORK}/stacks-blockchain/${STACKS_NETWORK}-stacks-blockchain-${STACKS_BLOCKCHAIN_VERSION}-${ARCHIVE_VERSION}.tar.gz"
213+
CHAINDATA_URL_SHA256="https://archive.hiro.so/${STACKS_NETWORK}/stacks-blockchain/${STACKS_NETWORK}-stacks-blockchain-${STACKS_BLOCKCHAIN_VERSION}-${ARCHIVE_VERSION}.sha256"
214+
215+
# define local storage locations
216+
PGDUMP_DEST="/tmp/stacks-blockchain-api-pg-${PG_VERSION}-${LOCAL_STACKS_API_VERSION}-${ARCHIVE_VERSION}.dump"
217+
PGDUMP_DEST_SHA256="/tmp/stacks-blockchain-api-pg-${PG_VERSION}-${LOCAL_STACKS_API_VERSION}-${ARCHIVE_VERSION}.sha256"
218+
CHAINDATA_DEST="/tmp/${STACKS_NETWORK}-stacks-blockchain-${STACKS_BLOCKCHAIN_VERSION}-${ARCHIVE_VERSION}.tar.gz"
219+
CHAINDATA_DEST_SHA256="/tmp/${STACKS_NETWORK}-stacks-blockchain-${STACKS_BLOCKCHAIN_VERSION}-${ARCHIVE_VERSION}.sha256"
220+
221+
exit_error() {
222+
echo "${1}"
223+
exit 1
224+
}
225+
226+
download_file(){
227+
# download the archive file if it exists. if not, exit with error
228+
local url=${1}
229+
local dest=${2}
230+
# retrieve http code of archive file
231+
local http_code=$(curl --output /dev/null --silent --head -w "%{http_code}" ${url})
232+
# if file does noe exist, exit
233+
if [[ "${http_code}" && "${http_code}" != "200" ]];then
234+
exit_error "Error ${url} not found"
235+
fi
236+
echo "Downloading ${url} data to: ${dest}"
237+
curl -L -# ${url} -o "${dest}" || exit_error "Error downloading ${url} to ${dest}"
238+
return 0
239+
}
240+
241+
verify_checksum(){
242+
# compares local sha256sum with downloaded sha256sum file
243+
local local_file=${1}
244+
local local_sha256=${2}
245+
local sha256=$(cat ${local_sha256} | awk {'print $1'} )
246+
local basename=$(basename ${local_file})
247+
local sha256sum=$(sha256sum ${local_file} | awk {'print $1'})
248+
# if sha256sum does not match file, exit
249+
if [ "${sha256}" != "${sha256sum}" ]; then
250+
exit_error "Error sha256 mismatch for ${basename}"
251+
fi
252+
return 0
253+
}
254+
255+
# download the pg_dump archive and verify the sha256sum matches
256+
download_file ${PGDUMP_URL} ${PGDUMP_DEST}
257+
download_file ${PGDUMP_URL_SHA256} ${PGDUMP_DEST_SHA256}
258+
verify_checksum ${PGDUMP_DEST} ${PGDUMP_DEST_SHA256}
259+
260+
# download the chainstate archive and verify the sha256sum matches
261+
download_file ${CHAINDATA_URL} ${CHAINDATA_DEST}
262+
download_file ${CHAINDATA_URL_SHA256} ${CHAINDATA_DEST_SHA256}
263+
verify_checksum ${CHAINDATA_DEST} ${CHAINDATA_DEST_SHA256}
264+
265+
# restore the pg_dump
266+
psql -U ${PG_USER} -c "alter user ${PG_USER} set max_parallel_workers_per_gather=0;" || exit_error "error altering user"
267+
pg_restore --username ${PG_USER} --verbose --create --dbname postgres ${PGDUMP_DEST} || exit_error "Error restoring API pg_dump data"
268+
psql -U ${PG_USER} -c "ALTER USER ${PG_USER} PASSWORD '${PG_PASSWORD}';" || exit_error "Error setting PG_USER password"
269+
psql -U ${PG_USER} -c "GRANT ALL PRIVILEGES ON DATABASE $PG_DATABASE TO $PG_USER;" || exit_error "error granting PG_USER privileges"
270+
271+
# update stacks-blockchain config to use extracted data, then extract it
272+
sed -i -e "s|^# working_dir.*|working_dir = \"${STACKS_BLOCKCHAIN_DIR}/data\"|;" ${STACKS_BLOCKCHAIN_DIR}/${STACKS_NETWORK}-follower-conf.toml || exit_error "error updating stacks-blockchain config"
273+
echo "Extracting stacks-blockchain chainstate data to: ${STACKS_BLOCKCHAIN_DIR}/data"
274+
tar -xvf "${CHAINDATA_DEST}" -C "${STACKS_BLOCKCHAIN_DIR}/data" || exit_error "Error extracting stacks-blockchain chainstate data"
275+
276+
# remove downloaded files to reduce disk usage
277+
if [ -f ${PGDUMP_DEST} ]; then
278+
rm -f ${PGDUMP_DEST} || exit_error "Error removing ${PGDUMP_DEST}"
279+
fi
280+
if [ -f ${PGDUMP_DEST_SHA256} ]; then
281+
rm -f ${PGDUMP_DEST_SHA256} || exit_error "Error removing ${PGDUMP_DEST_SHA256}"
282+
fi
283+
if [ -f ${CHAINDATA_DEST} ]; then
284+
rm -f ${CHAINDATA_DEST} || exit_error "Error removing ${CHAINDATA_DEST}"
285+
fi
286+
if [ -f ${CHAINDATA_DEST_SHA256} ]; then
287+
rm -f ${CHAINDATA_DEST_SHA256} || exit_error "Error removing ${CHAINDATA_DEST_SHA256}"
288+
fi
289+
exit 0
290+
EOM
291+
chmod +x /scripts/seed-chainstate.sh
292+
EOF
293+
294+
295+
296+
EXPOSE ${STACKS_BLOCKCHAIN_API_PORT} ${STACKS_CORE_RPC_PORT} ${STACKS_CORE_P2P_PORT}
297+
VOLUME /data
298+
CMD ["/entrypoint.sh"]
299+

0 commit comments

Comments
 (0)