fsi::Indexer Class Reference

#include <FSI_Indexer.hpp>

Collaboration diagram for fsi::Indexer:

Public Member Functions
	Indexer (const std::string &id, const bool threadsImpl=false)
	~Indexer ()
IndexerError	addExtendedInfo (const IndexerInfo &info)
	Add the exact path and the sub-paths to have the info from;.
IndexerError	addInfo (const IndexerInfo &info)
	Add the exact path to have the info from.
IndexerInfo	findIndex (const std::string &toFind)
	Find the index from the vector database from the indexer.
utils::TimeUtils_DateTime	getFileDTInfo (const std::string &path)
	Get when the file, dir or symlink was last modified.
std::string	getID () const
	Get this object's ID.
std::vector< IndexerInfo >	getIndexerInfo () const
	Return indexer information from this object.
std::vector< std::string >	getIndexPaths () const
	Get all the indexed paths from the vector indexes.
IndexerError	removeInfo (const std::string &searcher)
	Remove information from its ID or full path.
std::string	searchExactMatching (const std::string &find)
	Search for an exact matching path or ID;.
std::vector< std::string >	searchMatching (const std::string &path)
	Either use threaded or unthreaded search matching.

Protected Member Functions
virtual IndexerError	__addExtendedInfoStandard (const IndexerInfo &info)
	Standard non-threaded extended information.
virtual IndexerError	__addExtendedInfoThreaded (const IndexerInfo &info)
	Threaded extended information.
virtual IndexerPathType	__getPathType (const std::string &path) const
	Get if path from function param is a:
virtual std::vector< std::string >	__iteratePath (const std::string &path)
	Iterate all the files and directories (symlinks too) from path.
virtual std::vector< std::string >	__searchMatchingStandard (const std::string &find)
	Search all matching path or ID that contains path.
virtual std::vector< std::string >	__searchMatchingThreaded (const std::string &find)
	Search all matching path or ID that contains path using parallelization with multiple CPU threads.

Static Protected Member Functions
static std::vector< std::vector< std::string > >	__splitPathByCores (const std::vector< std::string > &paths)
	Split the paths found to be used in the future in different CPU cores.

Private Attributes
std::string	id
std::vector< IndexerInfo >	indexerInfo
bool	threadsImpl

Detailed Description

Definition at line 60 of file FSI_Indexer.hpp.

Constructor & Destructor Documentation

Indexer()

fsi::Indexer::Indexer	(	const std::string &	id,
		const bool	threadsImpl = false )

Definition at line 167 of file FSI_Indexer.cpp.

        : id(id), threadsImpl(threadsImpl)
    { }

References Indexer(), id, and threadsImpl.

Referenced by Indexer().

Here is the call graph for this function:

Here is the caller graph for this function:

~Indexer()

fsi::Indexer::~Indexer

(

)

Definition at line 172 of file FSI_Indexer.cpp.

    { }

Member Function Documentation

__addExtendedInfoStandard()

IndexerError fsi::Indexer::__addExtendedInfoStandard ( const IndexerInfo & info )

protectedvirtual

Standard non-threaded extended information.

• This will not add multiple extended information to be processed

  Parameters

  info	Get the information from the exact path and deeper

  Returns

  IndexerError Return error or success from IndexerError type

Definition at line 69 of file FSI_Indexer.cpp.

    {
        IndexerError error = this->addInfo(info);
        IndexerInfo totalInfo = info;
 
        if (__FSI_INDEXERERR_CHECK(error))
            return error;
 
        const std::vector<std::string> iteratedPaths = this->__iteratePath(info.path);
 
        for (const auto &x : iteratedPaths)
        {
            totalInfo.pathType = this->__getPathType(totalInfo.path);
            totalInfo.path = x;
 
            this->indexerInfo.emplace_back(totalInfo);
        }
 
        return error;
    }

References __addExtendedInfoStandard(), __FSI_INDEXERERR_CHECK, __getPathType(), __iteratePath(), addInfo(), indexerInfo, fsi::IndexerInfo::path, and fsi::IndexerInfo::pathType.

Referenced by __addExtendedInfoStandard(), and addExtendedInfo().

Here is the call graph for this function:

Here is the caller graph for this function:

__addExtendedInfoThreaded()

IndexerError fsi::Indexer::__addExtendedInfoThreaded ( const IndexerInfo & info )

protectedvirtual

Threaded extended information.

• This will add multiple extended information to be processed in the future

  Parameters

  info	Get the information from the exact path and deeper

  Returns

  IndexerError Return error or success from IndexerError type

Definition at line 90 of file FSI_Indexer.cpp.

    {
        IndexerError error = this->addInfo(info);
 
        if (__FSI_INDEXERERR_CHECK(error))
            return error;
 
        const auto paths = this->__iteratePath(info.path);
        const auto &chunks = this->__splitPathByCores(paths);
 
        std::vector<std::future<std::vector<IndexerInfo>>> tasks;
 
        for (auto &chunk : chunks)
        {
            if (chunk.empty())
                continue;
 
            tasks.emplace_back(std::async(std::launch::async,
                        [&, chunk]() -> std::vector<IndexerInfo>
            {
                std::vector<IndexerInfo> local;
 
                local.reserve(chunk.size());
 
                for (const auto &p : chunk)
                {
                    IndexerInfo tmp = info;
                    tmp.path = p;
                    tmp.pathType = this->__getPathType(p);
                    local.emplace_back(std::move(tmp));
                }
 
                return local;
            }));
        }
 
        // Merging phase
        for (auto &t : tasks)
        {
            auto local = t.get();
 
            this->indexerInfo.insert(
                    this->indexerInfo.end(),
                    std::make_move_iterator(local.begin()),
                    std::make_move_iterator(local.end()));
        }
 
        return error;
    }

References __addExtendedInfoThreaded(), __FSI_INDEXERERR_CHECK, __getPathType(), __iteratePath(), __splitPathByCores(), addInfo(), indexerInfo, fsi::IndexerInfo::path, and fsi::IndexerInfo::pathType.

Referenced by __addExtendedInfoThreaded(), and addExtendedInfo().

Here is the call graph for this function:

Here is the caller graph for this function:

__getPathType()

IndexerPathType fsi::Indexer::__getPathType ( const std::string & path ) const

protectedvirtual

Get if path from function param is a:

• File
• Dir
• SymLink
• Others

  Parameters

  path	Path to get the type from

  Returns

  IndexerPathType The type of the function param

Definition at line 20 of file FSI_Indexer.cpp.

    {
        if (fs::is_directory(path)) return IndexerPathType::Directory;
        else if (fs::is_symlink(path)) return IndexerPathType::SymLink;
 
        return IndexerPathType::File;
    }

References fsi::Directory, fsi::File, and fsi::SymLink.

Referenced by __addExtendedInfoStandard(), __addExtendedInfoThreaded(), and addInfo().

Here is the caller graph for this function:

__iteratePath()

std::vector< std::string > fsi::Indexer::__iteratePath ( const std::string & path )

protectedvirtual

Iterate all the files and directories (symlinks too) from path.

Parameters

path	Path to iterate

Returns

A tree of full path in a vector of strings

Definition at line 140 of file FSI_Indexer.cpp.

    {
        std::vector<std::string> paths;
 
        CVEC tmpPathsVec = cvec_init(-1, sizeof(char*));
 
        fsi_walk(&tmpPathsVec, path.c_str());
 
        for (size_t i = 0; i < tmpPathsVec.size; ++i)
        {
            char* p = *(char**)cvec_get(&tmpPathsVec, i);
 
            if (p)
            {
                paths.emplace_back(p, strlen(p));
 
                FSI_FREE(p);
            }
        }
 
        cvec_destroy(&tmpPathsVec);
 
        return paths;
    }

References __iteratePath(), FSI_FREE, and fsi_walk().

Referenced by __addExtendedInfoStandard(), __addExtendedInfoThreaded(), and __iteratePath().

Here is the call graph for this function:

Here is the caller graph for this function:

__searchMatchingStandard()

std::vector< std::string > fsi::Indexer::__searchMatchingStandard ( const std::string & find )

protectedvirtual

Search all matching path or ID that contains path.

Parameters

path	Path or ID to search

Returns

A vector of all the found paths

Definition at line 28 of file FSI_Indexer.cpp.

    {
        if (find.empty())
            return {};
 
        std::vector<std::string> vec;
 
        for (const auto &x : this->indexerInfo)
        {
            if (x.path.find(find) != std::string::npos)
                vec.emplace_back(x.path);
        }
 
        return vec;
    }

References indexerInfo.

Referenced by searchMatching().

Here is the caller graph for this function:

__searchMatchingThreaded()

std::vector< std::string > fsi::Indexer::__searchMatchingThreaded ( const std::string & find )

protectedvirtual

Search all matching path or ID that contains path using parallelization with multiple CPU threads.

Parameters

path	Path or ID to search

Returns

A vector of all the found paths

Definition at line 44 of file FSI_Indexer.cpp.

    {
        if (find.empty())
            return {};
 
        std::vector<std::string> result;
        std::mutex mutex;
 
        std::for_each(
                std::execution::par,
                indexerInfo.begin(),
                indexerInfo.end(),
                [&](const auto& x)
            {
                if (x.path.find(find) != std::string::npos)
                {
                    std::lock_guard<std::mutex> lock(mutex);
                    result.emplace_back(x.path);
                }
            }
        );
 
        return result;
    }

References indexerInfo.

Referenced by searchMatching().

Here is the caller graph for this function:

__splitPathByCores()

std::vector< std::vector< std::string > > fsi::Indexer::__splitPathByCores ( const std::vector< std::string > & paths )

inlinestaticprotected

Split the paths found to be used in the future in different CPU cores.

• Split the vector in to tiny vectors depending on how many CPU cores the hardware has

  Parameters

  paths

  The paths to split according to CPU cores

  Returns

  Return the splitted paths

Definition at line 124 of file FSI_Indexer.hpp.

        {
            const unsigned int threads = std::max(1u, std::thread::hardware_concurrency());
 
            std::vector<std::vector<std::string>> chunks(threads);
 
            for (size_t i = 0; i < paths.size(); ++i)
                chunks[i % threads].push_back(paths[i]);
 
            return chunks;
        }

Referenced by __addExtendedInfoThreaded().

Here is the caller graph for this function:

addExtendedInfo()

IndexerError fsi::Indexer::addExtendedInfo

(

const IndexerInfo &

info

)

Add the exact path and the sub-paths to have the info from;.

• Use Indexer::__addExtendedInfoStandard() or Indexer::__addExtendedInfoThreaded() if threads are disabled or enabled

  Parameters

  info	Get the information from the exact path and deeper

  Returns

  IndexerError Return error or success from IndexerError type

Definition at line 175 of file FSI_Indexer.cpp.

    {
        if (this->threadsImpl)
            return this->__addExtendedInfoThreaded(info);
        
        return this->__addExtendedInfoStandard(info);
    }

References __addExtendedInfoStandard(), __addExtendedInfoThreaded(), and threadsImpl.

Referenced by main().

Here is the call graph for this function:

Here is the caller graph for this function:

addInfo()

IndexerError fsi::Indexer::addInfo

(

const IndexerInfo &

info

)

Add the exact path to have the info from.

Parameters

info	Get the information from the exact path

Returns

IndexerError Return error or success from IndexerError type

Definition at line 198 of file FSI_Indexer.cpp.

    {
        IndexerError error;
        IndexerInfo totalInfo = info;
 
        // Do checks before adding
        if (totalInfo.path.empty())
        {
            error.code = fsi::codes::ERROR_ADDITION_FAIL;
            error.message = "Path is empty";
            error.fatal = false;
            error.raise = false;
 
            return error;
        }
 
        if (!fs::exists(totalInfo.path))
        {
            error.code = fsi::codes::ERROR_INVALID_PATH;
            error.message = ciof::format("Unknown path `%1`; No such file or directory exists", totalInfo.path);
            error.fatal = true;
 
            return error;
        }
 
        
        // Set path type
        totalInfo.pathType = this->__getPathType(totalInfo.path);
 
        // Add final information
 
        this->indexerInfo.emplace_back(totalInfo);
 
        return error;
    }

References __getPathType(), fsi::IndexerError::code, fsi::codes::ERROR_ADDITION_FAIL, fsi::codes::ERROR_INVALID_PATH, fsi::IndexerError::fatal, indexerInfo, fsi::IndexerError::message, fsi::IndexerInfo::path, fsi::IndexerInfo::pathType, and fsi::IndexerError::raise.

Referenced by __addExtendedInfoStandard(), and __addExtendedInfoThreaded().

Here is the call graph for this function:

Here is the caller graph for this function:

findIndex()

IndexerInfo fsi::Indexer::findIndex

(

const std::string &

toFind

)

Find the index from the vector database from the indexer.

Parameters

toFind

Search for the path or ID and remove them from the indexer

Returns

IndexerInfo Find the indexed info from the indexer "data base"

Definition at line 234 of file FSI_Indexer.cpp.

    {
        for (const auto &x : this->indexerInfo)
        {
            if (x.path == toFind || x.id == toFind)
                return x;
        }
 
        return { };
    }

References indexerInfo.

Referenced by main(), and searchExactMatching().

Here is the caller graph for this function:

getFileDTInfo()

utils::TimeUtils_DateTime fsi::Indexer::getFileDTInfo

(

const std::string &

path

)

Get when the file, dir or symlink was last modified.

Parameters

path	Path to get date time information

Returns

Date time info

Definition at line 275 of file FSI_Indexer.cpp.

    {
        const fs::file_time_type &lastWriteTime = fs::last_write_time(path);
 
        return utils::fsClockDataToDT(lastWriteTime);
    }

References fsi::utils::fsClockDataToDT().

Referenced by main().

Here is the call graph for this function:

Here is the caller graph for this function:

getID()

std::string fsi::Indexer::getID

(

)

const

Get this object's ID.

Returns

Object ID

Definition at line 295 of file FSI_Indexer.cpp.

    { return this->id; }

References id.

getIndexerInfo()

std::vector< IndexerInfo > fsi::Indexer::getIndexerInfo

(

)

const

Return indexer information from this object.

Returns

Get all the indexed information that was added

Definition at line 282 of file FSI_Indexer.cpp.

    { return this->indexerInfo; }

References indexerInfo.

getIndexPaths()

std::vector< std::string > fsi::Indexer::getIndexPaths

(

)

const

Get all the indexed paths from the vector indexes.

Returns

All the indexed paths in a vector

Definition at line 285 of file FSI_Indexer.cpp.

    {
        std::vector<std::string> additions;
 
        for (const auto &x : this->indexerInfo)
        { additions.emplace_back(x.path); }
 
        return additions;
    }

References indexerInfo.

removeInfo()

IndexerError fsi::Indexer::removeInfo

(

const std::string &

searcher

)

Remove information from its ID or full path.

Parameters

searcher

Search for the path or ID and remove them from the indexer

Returns

IndexerError Return error or success from IndexerError type

Definition at line 245 of file FSI_Indexer.cpp.

    {
        IndexerError error;
 
        // Do checks
 
        if (searcher.empty())
        {
            error.code = fsi::codes::ERROR_EMPTY_VALUE;
            error.message = "Empty value to remove from the Indexer Information list";
            error.fatal = false;
            error.raise = false;
 
            return error;
        }
 
        // Erase the found object (IndexerInfo) value by ID or Path
        this->indexerInfo.erase(
                std::remove_if(
                    this->indexerInfo.begin(),
                    this->indexerInfo.end(),
                    [&](const IndexerInfo &info) {
                        return (info.id == searcher || info.path == searcher);
                    }),
                this->indexerInfo.end()
        );
 
        return error;
    }

References fsi::IndexerError::code, fsi::codes::ERROR_EMPTY_VALUE, fsi::IndexerError::fatal, indexerInfo, fsi::IndexerError::message, and fsi::IndexerError::raise.

searchExactMatching()

std::string fsi::Indexer::searchExactMatching

(

const std::string &

find

)

Search for an exact matching path or ID;.

Parameters

find	Get the ID or path to find

Returns

The first match found

Definition at line 183 of file FSI_Indexer.cpp.

    {
        if (find.empty())
            return "";
 
        return this->findIndex(find).path;
    }

References findIndex(), and fsi::IndexerInfo::path.

Here is the call graph for this function:

searchMatching()

std::vector< std::string > fsi::Indexer::searchMatching

(

const std::string &

path

)

Either use threaded or unthreaded search matching.

• Use Indexer::__searchMatchingThreaded() or Indexer::__searchMatchingStandard()

  Parameters

  path	Path to read the data to search

  Returns

  All the data found in a vector of strings

Definition at line 191 of file FSI_Indexer.cpp.

    {
        if (this->threadsImpl) return this->__searchMatchingThreaded(find);
 
        return this->__searchMatchingStandard(find);
    }

References __searchMatchingStandard(), __searchMatchingThreaded(), and threadsImpl.

Referenced by main().

Here is the call graph for this function:

Here is the caller graph for this function:

Member Data Documentation

id

std::string fsi::Indexer::id

private

Definition at line 63 of file FSI_Indexer.hpp.

Referenced by Indexer(), and getID().

indexerInfo

std::vector<IndexerInfo> fsi::Indexer::indexerInfo

private

Definition at line 65 of file FSI_Indexer.hpp.

Referenced by __addExtendedInfoStandard(), __addExtendedInfoThreaded(), __searchMatchingStandard(), __searchMatchingThreaded(), addInfo(), findIndex(), getIndexerInfo(), getIndexPaths(), and removeInfo().

threadsImpl

bool fsi::Indexer::threadsImpl

private

Definition at line 67 of file FSI_Indexer.hpp.

Referenced by Indexer(), addExtendedInfo(), and searchMatching().

---

The documentation for this class was generated from the following files:

• inc/fsi/lib/FSI_Indexer.hpp
• src/lib/FSI_Indexer.cpp