Rippled Audit

The App Submodule

The app submodule resides in src/ripple/app directory and is reponsible for the highest level of structured functionality in the application. The main entry point in rippled instantiates the AppImp class, which is an internal wrapper around the Application interface.

src/ripple/app/main/Application.cpp

2224
2225
2226
2227
2228
2229
2230
2231
2232
2233
std::unique_ptr<Application>
make_Application (
    std::unique_ptr<Config> config,
    std::unique_ptr<Logs> logs,
    std::unique_ptr<TimeKeeper> timeKeeper)
{
    return std::make_unique<ApplicationImp> (
        std::move(config), std::move(logs),
            std::move(timeKeeper));
}

Application Interface (src/ripple/app/main/Application.h)

83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
127
128
129
130
131
132
133
134
135
136
137
138
139
140
141
142
143
144
145
146
147
148
149
150
151
152
153
154
155
156
157
158
159
160
161
162
163
164
165
166
167
168
169
170
171
172
173
174
175
176
177
178
179
180
181
182
183
class Application : public beast::PropertyStream::Source
{
public:
    // comment removed for brievity
    using MutexType = std::recursive_mutex;
    virtual MutexType& getMasterMutex () = 0;

public:
    Application ();

    virtual ~Application () = default;

    virtual bool setup() = 0;
    virtual void doStart(bool withTimers) = 0;
    virtual void run() = 0;
    virtual bool isShutdown () = 0;
    virtual void signalStop () = 0;
    virtual bool checkSigs() const = 0;
    virtual void checkSigs(bool) = 0;

    //
    // ---
    //

    virtual Logs&                   logs() = 0;
    virtual Config&                 config() = 0;

    virtual
    boost::asio::io_service&
    getIOService () = 0;

    virtual CollectorManager&           getCollectorManager () = 0;
    virtual Family&                     family() = 0;
    virtual Family*                     shardFamily() = 0;
    virtual TimeKeeper&                 timeKeeper() = 0;
    virtual JobQueue&                   getJobQueue () = 0;
    virtual NodeCache&                  getTempNodeCache () = 0;
    virtual CachedSLEs&                 cachedSLEs() = 0;
    virtual AmendmentTable&             getAmendmentTable() = 0;
    virtual HashRouter&                 getHashRouter () = 0;
    virtual LoadFeeTrack&               getFeeTrack () = 0;
    virtual LoadManager&                getLoadManager () = 0;
    virtual Overlay&                    overlay () = 0;
    virtual TxQ&                        getTxQ() = 0;
    virtual ValidatorList&              validators () = 0;
    virtual ValidatorSite&              validatorSites () = 0;
    virtual ManifestCache&              validatorManifests () = 0;
    virtual ManifestCache&              publisherManifests () = 0;
    virtual Cluster&                    cluster () = 0;
    virtual RCLValidations&             getValidations () = 0;
    virtual NodeStore::Database&        getNodeStore () = 0;
    virtual NodeStore::DatabaseShard*   getShardStore() = 0;
    virtual InboundLedgers&             getInboundLedgers () = 0;
    virtual InboundTransactions&        getInboundTransactions () = 0;

    virtual
    TaggedCache <uint256, AcceptedLedger>&
    getAcceptedLedgerCache () = 0;

    virtual LedgerMaster&           getLedgerMaster () = 0;
    virtual NetworkOPs&             getOPs () = 0;
    virtual OrderBookDB&            getOrderBookDB () = 0;
    virtual TransactionMaster&      getMasterTransaction () = 0;
    virtual perf::PerfLog&          getPerfLog () = 0;

    virtual
    std::pair<PublicKey, SecretKey> const&
    nodeIdentity () = 0;

    virtual
    PublicKey const &
    getValidationPublicKey() const  = 0;

    virtual Resource::Manager&      getResourceManager () = 0;
    virtual PathRequests&           getPathRequests () = 0;
    virtual SHAMapStore&            getSHAMapStore () = 0;
    virtual PendingSaves&           pendingSaves() = 0;
    virtual AccountIDCache const&   accountIDCache() const = 0;
    virtual OpenLedger&             openLedger() = 0;
    virtual OpenLedger const&       openLedger() const = 0;
    virtual DatabaseCon&            getTxnDB () = 0;
    virtual DatabaseCon&            getLedgerDB () = 0;

    virtual
    std::chrono::milliseconds
    getIOLatency () = 0;

    virtual bool serverOkay (std::string& reason) = 0;

    virtual beast::Journal journal (std::string const& name) = 0;

    /* Returns the number of file descriptors the application wants */
    virtual int fdlimit () const = 0;

    /** Retrieve the "wallet database" */
    virtual DatabaseCon& getWalletDB () = 0;

    /** Ensure that a newly-started validator does not sign proposals older
     * than the last ledger it persisted. */
    virtual LedgerIndex getMaxDisallowedLedger() = 0;
};

Many ApplicationImp members instantiated through the class constructor are passed a reference to this (the pointer to the Application object) which is used to access other components / subsystems in the Application heirarchy. As seen above, accessors on the abstract Application interface are invoked to retrieve references to central modules instantiated in the scope of Application.

The ApplicationImp class is the primary concrete centralization point, deriving from the core module's RootStoppable class, facilitating the scheduling of its operations and those of its children.

ApplicationImp (src/ripple/app/main/Application.cpp)

237
238
239
240
241
242
243
244
245
246
247
248
249
250
251
252
253
254
255
256
257
258
259
260
261
262
263
264
265
266
267
268
269
270
271
272
273
274
275
276
277
278
279
280
281
282
283
284
285
286
287
288
289
290
291
292
293
294
295
296
297
298
299
300
301
302
303
304
305
306
307
308
309
310
311
312
313
314
    class ApplicationImp
        : public Application
        , public RootStoppable
        , public BasicApp
    {
       // ...skiped internal class for brevity
       public:
           std::unique_ptr<Config> config_;
           std::unique_ptr<Logs> logs_;
           std::unique_ptr<TimeKeeper> timeKeeper_;

           beast::Journal m_journal;
           std::unique_ptr<perf::PerfLog> perfLog_;
           Application::MutexType m_masterMutex;

           // Required by the SHAMapStore
           TransactionMaster m_txMaster;

           NodeStoreScheduler m_nodeStoreScheduler;
           std::unique_ptr <SHAMapStore> m_shaMapStore;
           PendingSaves pendingSaves_;
           AccountIDCache accountIDCache_;
           boost::optional<OpenLedger> openLedger_;

           // These are not Stoppable-derived
           NodeCache m_tempNodeCache;
           std::unique_ptr <CollectorManager> m_collectorManager;
           CachedSLEs cachedSLEs_;
           std::pair<PublicKey, SecretKey> nodeIdentity_;
           ValidatorKeys const validatorKeys_;

           std::unique_ptr <Resource::Manager> m_resourceManager;

           // These are Stoppable-related
           std::unique_ptr <JobQueue> m_jobQueue;
           std::unique_ptr <NodeStore::Database> m_nodeStore;
           std::unique_ptr <NodeStore::DatabaseShard> shardStore_;
           detail::AppFamily family_;
           std::unique_ptr <detail::AppFamily> sFamily_;
           // VFALCO TODO Make OrderBookDB abstract
           OrderBookDB m_orderBookDB;
           std::unique_ptr <PathRequests> m_pathRequests;
           std::unique_ptr <LedgerMaster> m_ledgerMaster;
           std::unique_ptr <InboundLedgers> m_inboundLedgers;
           std::unique_ptr <InboundTransactions> m_inboundTransactions;
           TaggedCache <uint256, AcceptedLedger> m_acceptedLedgerCache;
           std::unique_ptr <NetworkOPs> m_networkOPs;
           std::unique_ptr <Cluster> cluster_;
           std::unique_ptr <ManifestCache> validatorManifests_;
           std::unique_ptr <ManifestCache> publisherManifests_;
           std::unique_ptr <ValidatorList> validators_;
           std::unique_ptr <ValidatorSite> validatorSites_;
           std::unique_ptr <ServerHandler> serverHandler_;
           std::unique_ptr <AmendmentTable> m_amendmentTable;
           std::unique_ptr <LoadFeeTrack> mFeeTrack;
           std::unique_ptr <HashRouter> mHashRouter;
           RCLValidations mValidations;
           std::unique_ptr <LoadManager> m_loadManager;
           std::unique_ptr <TxQ> txQ_;
           ClosureCounter<void, boost::system::error_code const&> waitHandlerCounter_;
           boost::asio::steady_timer sweepTimer_;
           boost::asio::steady_timer entropyTimer_;
           bool startTimers_;

           std::unique_ptr <DatabaseCon> mTxnDB;
           std::unique_ptr <DatabaseCon> mLedgerDB;
           std::unique_ptr <DatabaseCon> mWalletDB;
           std::unique_ptr <Overlay> m_overlay;
           std::vector <std::unique_ptr<Stoppable>> websocketServers_;

           boost::asio::signal_set m_signals;
           beast::WaitableEvent m_stop;

           std::atomic<bool> checkSigs_;

           std::unique_ptr <ResolverAsio> m_resolver;

           io_latency_sampler m_io_latency_sampler;

ApplicationImp instantiates various members with its local reference, so that members deriving from Stoppable can be registered as children of ApplicationImp. This results in a component heirarchy whose component lifecycle can be managed as a whole, being started and stopped at the necessary times in the correct order.

Stoppable (src/ripple/core/Stoppable.h)

237
238
239
240
241
242
243
244
245
246
247
248
249
250
251
252
253
254
255
256
257
258
259
260
261
262
263
264
265
266
267
268
269
270
271
272
273
274
275
276
277
278
279
280
281
282
283
284
285
286
287
288
289
290
291
292
293
294
295
296
297
298
299
300
301
302
303
304
305
306
307
308
309
310
311
312
313
314
315
316
317
318
319
    class Stoppable
    {
    protected:
        Stoppable (std::string name, RootStoppable& root);

    public:
        /** Create the Stoppable. */
        Stoppable (std::string name, Stoppable& parent);

        /** Destroy the Stoppable. */
        virtual ~Stoppable ();

        /** Returns `true` if the stoppable should stop. */
        bool isStopping () const;

        /** Returns `true` if the requested stop has completed. */
        bool isStopped () const;

        /** Returns `true` if all children have stopped. */
        bool areChildrenStopped () const;

        /* JobQueue uses this method for Job counting. */
        inline JobCounter& jobCounter ();

        /** Sleep or wake up on stop.
            @return `true` if we are stopping
        */
        bool
        alertable_sleep_until(
            std::chrono::system_clock::time_point const& t);

    protected:
        /** Called by derived classes to indicate that the stoppable has stopped. */
        void stopped ();

    private:
        /** Override called during preparation.
            Since all other Stoppable objects in the tree have already been
            constructed, this provides an opportunity to perform initialization which
            depends on calling into other Stoppable objects.
            This call is made on the same thread that called prepare().
            The default implementation does nothing.
            Guaranteed to only be called once.
        */
        virtual void onPrepare ();

        /** Override called during start. */
        virtual void onStart ();

        /** Override called when the stop notification is issued.
            The call is made on an unspecified, implementation-specific thread.
            onStop and onChildrenStopped will never be called concurrently, across
            all Stoppable objects descended from the same root, inclusive of the
            root.
            It is safe to call isStopping, isStopped, and  areChildrenStopped from
            within this function; The values returned will always be valid and never
            change during the callback.
            The default implementation simply calls stopped(). This is applicable
            when the Stoppable has a trivial stop operation (or no stop operation),
            and we are merely using the Stoppable API to position it as a dependency
            of some parent service.
            Thread safety:
                May not block for long periods.
                Guaranteed only to be called once.
                Must be safe to call from any thread at any time.
        */
        virtual void onStop ();

        /** Override called when all children have stopped.
            The call is made on an unspecified, implementation-specific thread.
            onStop and onChildrenStopped will never be called concurrently, across
            all Stoppable objects descended from the same root, inclusive of the
            root.
            It is safe to call isStopping, isStopped, and  areChildrenStopped from
            within this function; The values returned will always be valid and never
            change during the callback.
            The default implementation does nothing.
            Thread safety:
                May not block for long periods.
                Guaranteed only to be called once.
                Must be safe to call from any thread at any time.
        */
        virtual void onChildrenStopped ();

Stoppable Constructor - Registering Child (src/ripple/core/impl/Stoppable.cpp)

32
33
34
35
36
37
38
39
40
41
    Stoppable::Stoppable (std::string name, Stoppable& parent)
        : m_name (std::move (name))
        , m_root (parent.m_root)
        , m_child (this)
    {
        // Must not have stopping parent.
        assert (! parent.isStopping());

        parent.m_children.push_front (&m_child);
    }

See the complete Stoppable lifecycle documentation here

ApplicationImp also derives from BasicApp which provides access to the centralized boost::asio::io_service instance, satisfying the core Input/Output functionality for asynchronous objects in the system (sockets, timers, etc). The io_service is instantiated with the class, along with a configurable-sized thread pool to satisfy I/O requests. This is all accomplished in the BasicApp constructor along with actually running the I/O service so that it is immediately available from the application start.

23
24
25
26
27
28
29
30
31
32
33
34
35
36
    BasicApp::BasicApp(std::size_t numberOfThreads)
    {
        work_.emplace (io_service_);
        threads_.reserve(numberOfThreads);
        while(numberOfThreads--)
            threads_.emplace_back(
                [this, numberOfThreads]()
                {
                    beast::setCurrentThreadName(
                        std::string("io_service #") +
                            std::to_string(numberOfThreads));
                    this->io_service_.run();
                });
    }

The Application class heirarchy can be seen below

After instantiating ApplicationImp, the main method invokes doStart() on it which invokes prepare and start() on the base RootStoppable interface. Both of these methods call the corresponding callbacks (onPrepare() and onStart()) on the class heirarchy recursively.

1459
1460
1461
1462
1463
1464
1465
    void
    ApplicationImp::doStart(bool withTimers)
    {
        startTimers_ = withTimers;
        prepare ();
        start ();
    }
171
172
173
174
175
176
177
178
179
180
    bool RootStoppable::isStopping() const
    {
        return m_calledStop;
    }

    void RootStoppable::prepare ()
    {
        if (m_prepared.exchange (true) == false)
            prepareRecursive ();
    }
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
    void Stoppable::prepareRecursive ()
    {
        for (Children::const_iterator iter (m_children.cbegin ());
            iter != m_children.cend(); ++iter)
            iter->stoppable->prepareRecursive ();
        onPrepare ();
    }

    void Stoppable::startRecursive ()
    {
        onStart ();
        for (Children::const_iterator iter (m_children.cbegin ());
            iter != m_children.cend(); ++iter)
            iter->stoppable->startRecursive ();
    }

Each component then initializes and starts operations, in the case of ApplicationImp, we see the Sweep and Entropy Timers are started (scheduling systemwide cleanup sweeps and the updating the entropy buffer respectively) before the asynchronous resolver is started (which rippled uses to lookup host and service names).

The entire Stoppable class heirarchy can be seen below. We will be exploring each of these modules going forward, diving into the functionality satisfied by each of these components and how it is achieved in more detail.

Finally the main method invokes ApplicationImp::run() which blocks on the m_stop WaitableEvent until it is released later on in the application's lifecycle (accomplished via a standard pthread mutex and wait condition).